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FOREWORD 


The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  has  been 
conducting  research  over  the  past  several  years  to  better  rmderstand  the  challenges  of  Military 
Operations  in  Urban  Terrain  (MOUT).  Such  operations  place  great  demands  on  military 
personnel  for  new  types  of  technical  skill.  They  also  generate  a  peculiar  set  of  cognitive 
requirements  related  to  a  wide  range  of  judgments  and  decisions  that  become  necessary  when 
military  forces  enter  urban  environments. 

Earlier  this  year  we  published  a  guide  for  the  warfighting  community  that  provided  an 
overview  of  the  decision  requirements  platoon  leaders  will  face  in  the  process  of  clearing 
buildings,  perhaps  the  most  procedurally  complicated  and  dangerous  of  all  MOUT  tasks  (ARI 
Research  Product  2001-02).  We  then  released  a  CD-based  instructor  training  program  called 
IMproving  Performance  through  Applied  Cognitive  Training  (IMPACT).  Appropriate  for  either 
institutional  or  unit  training,  IMPACT  enables  instructors  to  facilitate  their  own  Decision 
Making  Game  sessions,  to  create  their  own  Decision  Making  Game  scenarios,  and  to  help 
platoon  leaders  become  better  at  making  platoon-level  MOUT  decisions.  Recently,  we  published 
a  comprehensive  report  documenting  the  conceptual  foundation  of  both  the  warfighting  guide 
and  IMPACT  training  program  (ARI  Research  Report  1776). 

The  present  report  is  the  last  of  four  research  products  developed  by  Klein  Associates 
under  a  Small  Business  Innovation  Research  (SBIR)  contract  aimed  at  training  platoon  leaders  to 
make  more  accurate  and  timely  decisions  during  urban  operations.  It  documents  the  conduct  and 
findings  of  a  usability  and  training  effectiveness  evaluation  of  the  IMPACT  program  using 
cadets  and  tactical  instructors  at  the  U.S.  Military  Academy.  Results  were  encouraging. 
Specifically,  IMPACT  training  was  found  to  be  as  effective  as  more  traditional  after-action 
review  methods  of  instruction,  with  an  even  higher  level  of  apparent  usability.  Major  findings 
from  the  IMPACT  development  and  evaluation  process  were  presented  to  a  wide  variety  of 
personnel  within  the  U.S.  Army  Training  and  Doctrine  Command  (TRADOC-DCST)  tteough 
briefings  conducted  at  Fort  Monroe,  Virginia  in  April  2001  and  at  Fort  Leavenworth,  Kansas  in 
August  2001. 


TA  M.  SIMUTIS 
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EVALUATING  AN  APPROACH  TO  MOUT  DECISION  SKILLS  TRAINING 

EXECUTIVE  SUMMARY _ _ 


Research  Requirement: 

A  new  instructional  approach.  Decision  Skills  Training  (DST),  has  recently  been 
developed  to  improve  the  performance  of  military  decision  makers  by  boosting  their  domain 
expertise.  Traditional  approaches  to  decision  training  have  attempted  to  teach  generic,  anal5tic 
strategies  that  have  not  proven  effective  in  field  settings.  DST,  which  has  as  its  basis  the 
Recognition-Primed  Decision  model  (Klein,  1989;  1997)  and  the  research  literature  on  the 
development  of  expertise  (e.g.,  Chi,  Glaser,  &  Farr,  1988;  Ericsson  &  Chamess,  1994),  has 
received  positive  reviews  from  instructors  and  students  as  well  as  encouraging  anecdotal 
evidence  as  to  its  effectiveness. 

In  an  effort  to  improve  the  decision-making  performance  of  small  unit  leaders  in  Military 
Operations  in  Urban  Terrain  (MOUT),  we  first  tailored  a  DST  approach  at  the  platoon  level,  and 
then  developed  a  train-the-trainer  multimedia  CD-ROM  -  IMPACT  -  to  provide  instruction  to 
facilitators  wishing  to  deliver  the  MOUT  DST  training.  This  report  describes  a  preliminary 
evaluation  performed  at  the  U.S.  Military  Academy  (USMA)  to  evaluate  the  usability  of  the 
IMPACT  train-the-trainer  CD-ROM  and  to  explore  methodological  issues  associated  with 
assessing  decision  quality. 

Procedure: 

Twelve  U.S.  Army  captains  and  65  cadets  in  their  junior  year  at  the  USMA  were 
recruited  as  participants.  The  captains  facilitated  the  MOUT  training;  the  cadets  were  their 
trainees.  Half  of  the  cadets  were  assigned  to  the  experimental  group  and  were  facilitated  by 
IMPACT-trained  facilitators  (the  IMPACT  group).  The  other  half  were  assigned  to  the  control 
group  and  were  facilitated  by  captains  who  had  not  been  exposed  to  IMPACT  (the  NO-IMPACT 
group).  All  cadets  participated  in  a  series  of  three  55-minute  sessions.  The  first  two  sessions 
were  training  sessions  facilitated  by  the  captains.  Facilitators  from  both  the  IMPACT  and  NO¬ 
IMPACT  conditions  were  provided  with  two  MOUT  training  scenarios  called  Decision  Making 
Games  (DMGs).  One  DMG  was  to  be  used  per  training  session.  The  IMPACT-trained 
facilitators  were  instructed  to  use  the  techniques  described  in  the  IMPACT  CD  to  facilitate  and 
debrief  the  DMGs.  The  facilitators  in  the  NO-IMPACT  group  were  instructed  to  facilitate  the 
DMGs  and  then  use  a  standard  After- Action  Review  technique  to  debrief  the  DMGs.  Cadets 
completed  a  Session  Evaluation  Form  at  the  end  of  each  of  Ae  first  two  sessions.  The  third 
session  was  a  test  session  attended  only  by  cadets  and  led  by  a  member  of  the  research  team. 
Cadets  completed  three  response  forms,  which  served  as  dependent  measures:  a  DMG  response 
form,  a  MOUT  Knowledge  Test,  and  a  Cadet  Post-Test  Questionnaire.  Facilitators  were  also 
asked  to  complete  a  Facilitator  Post-Session  Questionnaire. 
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Findings: 


We  set  out  to  determine  whether  IMPACT  is  a  usable  train-the-trainer  tool,  and  whether 
we  could  demonstrate  a  measurable  difference  in  the  decision  making  performance  of  individuals 
who  participated  in  DMG  sessions  led  by  an  IMPACT-trained  facilitator  versus  individuals  who 
participated  in  DMG  sessions  led  by  facilitators  who  had  not  had  access  to  IMPACT.  The  results 
of  the  study  are  mixed.  It  appears  that  IMPACT  is  a  very  usable  training  tool,  but  this  particular 
study  was  unable  to  demonstrate  that  it  improves  the  decision-making  performance  of  trainees. 

In  addition,  important  methodological  advances  in  assessing  decision  quality  were  made. 

Captains  in  the  IMPACT  condition  judged  the  instructional  modules  within  IMPACT  to 
be  very  useful.  The  average  “usefulness”  rating  was  a  “4”  on  a  5-point  scale  where  5  was  labeled 
“extremely  useful”  and  1  was  labeled  “not  at  all  useful.”  All  of  the  facilitators  in  the  IMPACT 
conditions  reported  that  they  were  likely  to  incorporate  aspects  of  the  IMPACT  debrief  technique 
into  their  future  After-Action  Reviews. 

Cadet  responses  on  the  DMG  response  form  were  rated  by  independent  SMEs  who  were 
blind  to  condition.  An  examination  of  the  means  for  the  two  conditions  did  not  indicate  reliable 
differences  in  decision-making  performance  as  a  result  of  the  IMPACT  training.  Likewise  there 
were  no  significant  differences  in  performance  on  the  MOUT  Knowledge  Test.  The  Session 
Evaluation  Forms,  however,  indicate  that  for  three  out  of  four  training  sessions  facilitated  by  a 
captain,  the  cadets  in  the  IMPACT  group  had  more  positive  evaluations  than  the  NO-IMPACT 
group.  In  addition,  it  is  noteworthy  that  cadets  in  both  groups  responded  quite  favorably  to  the 
use  of  DMGs  in  training  sessions. 

A  notable  outcome  of  this  study  was  the  refinement  of  a  process  for  operationalizing 
measures  of  decision-making  quality.  We  have  previously  employed  SMEs  to  rate  the  quality  of 
participant  responses,  but  have  struggled  to  get  reliable  ratings  from  SMEs  who  often  have 
somewhat  different  ideas  about  what  constitutes  “good”  decision  making.  In  this  effort,  we 
developed  a  process  that  produced  high  inter-rater  reliabilities.  In  addition,  we  developed  an 
objective  measure  of  decision  quality  in  MOUT  settings  in  the  form  of  a  multiple-choice  test. 

Utilization  of  Findings: 

While  subjective  measures  indicated  favorable  reviews  of  the  IMPACT  CD  and  the 
IMPACT-facilitated  sessions,  we  were  not  able  to  show  significant  enhancement  in  decision¬ 
making  performance  as  a  result  of  the  IMPACT  training.  We  believe  the  fact  that  cadets  received 
only  limited  exposure  to  the  training  -  two  55-minute  sessions  -  is  the  primary  reason  for  the 
lack  of  significant  findings.  DST  is  not  meant  to  be  a  “quick  fix”  training  intervention;  it  is 
intended  as  a  long-term  approach.  Furthermore,  the  IMPACT  facilitators  were  able  to  spend  on 
average  only  130  minutes  using  the  CD  and  learning  the  approach.  They  were  not  provided 
practice  nor  feedback  on  their  use  of  the  DST  techniques  before  facilitating  the  sessions  with 
cadets.  This  is  another  critical  inhibitor  to  finding  performance  effects.  We  are  exploring 
possibilities  for  supplementing  IMPACT  with  opportunities  for  instructors  to  practice  the 
techniques  and  receive  feedback  on  their  implementation  of  the  DST  approach.  We  are  also 
pursuing  another  evaluation  study  that  resolves  the  limitations  of  the  current  study. 
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INTRODUCTION 


Klein  and  his  colleagues  (Klein,  1997;  Pliske,  McCloskey,  &  Klein,  2001)  developed  an 
instructional  approach,  called  Decision  Skills  Training  (DST),  that  improves  decision-making 
performance  by  boosting  the  decision  makers’  domain  expertise.  Traditional  training  approaches 
for  improving  decision-making  skills  have  attempted  to  teach  generic  strategies  such  as  decision 
analysis  in  which  the  decision  maker  analyzes  the  costs  and  benefits  of  alternative  courses  of 
action.  This  type  of  decision  training  has  not  been  proven  effective  in  improving  performance  in 
field  settings.  One  reason  for  the  inadequacy  of  this  anal5dical  approach  is  that  many  decision¬ 
making  situations  do  not  involve  choices  between  alternative  courses  of  action.  Previous 
research  has  shown  that  skilled  decision  makers  spend  more  time  sizing  up  the  situation  than 
comparing  alternative  cotirses  of  action.  These  decision  makers  are  able  to  quickly  generate  an 
acceptable  course  of  action  based  on  their  previous  experiences.  For  example,  Klein  (1989; 

1997)  presented  a  Recognition-Primed  Decision  model,  which  describes  how  people  use 
experience  to  make  rapid  decisions  under  conditions  of  time  pressure  and  uncertainty  that 
preclude  the  use  of  analytical  strategies. 

Klein  Associates’  DST  approach  is  based  on  the  Recognition-Primed  Decision  model  as 
well  as  findings  fi’om  a  review  of  the  research  literature  on  the  development  of  expertise  (e.g., 
(Chi  et  al.,  1988;  Ericsson  &  Chamess,  1994).  We  identified  several  strategies  that  experts  use  in 
order  to  learn  more  rapidly  and  effectively.  These  strategies  include:  engaging  in  deliberate 
practice,  so  that  each  opportunity  for  practice  has  a  goal  and  evaluation  criteria;  compiling  an 
extensive  experience  bank;  obtaining  feedback  that  is  accurate  and  diagnostic;  building  mental 
models;  developing  metacognitive  skills;  and  becoming  more  mindful  of  opportunities  for 
learning. 

We  have  developed  DST  programs  for  U.S.  Marine  Corps  officers  and  enlisted 
personnel,  U.S.  Navy  pilots,  urban  firefighters,  and  business  executives.  These  programs  were  all 
delivered  through  train-the-trainer  workshops  facilitated  by  a  live  instructor;  these  are  expensive 
and  can  only  reach  a  limited  ntimber  of  participants.  In  this  Phase  n  SBIR  effort  sponsored  by 
the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences,  a  DST  program  was 
developed  for  platoon  leaders  in  Military  Operations  in  Urban  Terrain  (MOUT)  building  clearing 
missions  (Phillips,  McCloskey,  McDermott,  Wiggins,  Battaglia,  &  Klein,  2001).  The  primary 
product  of  the  MOUT  DST  program  was  a  multimedia  train-the-trainer  CD-ROM  -  IMPACT 
(BMproving  Performance  through  Applied  Cognitive  Training)  -  that  teaches  Army  instructors  to 
implement  the  DST  with  lieutenants.  IMPACT  was  designed  to  address  the  specific  training 
requirements  for  platoon  leaders  in  MOUT,  and  to  do  so  in  a  way  that  luilike  train-the-trainer 
workshops,  could  reach  a  larger  instructor  audience  for  less  overall  cost.  This  report  describes  a 
preliminary  evaluation  study  of  IMPACT  conducted  at  the  U.S.  Military  Academy  (USMA)  at 
West  Point. 


Tailoring  a  DST  Approach  for  MOUT 


Our  DST  programs  typically  involve  a  variety  of  learning  tools  that  are  used  in 
conjunction  with  Decision  Making  Games  (DMGs).  DMGs  are  low-fidelity,  paper-and-pencil, 
simulations  of  incidents  that  might  occur  in  a  particular  domain  (Pliske,  McCloskey,  &  Klein, 
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2001 ;  Schmitt  1994).  Trainees  are  given  a  dilemma  in  which  a  decision  must  be  made,  and  are 
only  given  a  few  minutes  to  develop  their  course  of  action.  The  primary  learning  tool  associated 
with  the  DMGs  is  the  Decision  Making  Critique  (DMC),  which  is  a  debriefing  technique 
developed  to  supplement  the  standard  After  Action  Reviews  (AARs)  conducted  in  military 
training  settings.  The  DMC  consists  of  a  series  of  questions  designed  to  focus  the  group 
discussion  on  critical  decisions,  the  important  cues  involved  in  assessing  the  situation,  and  the 
types  of  uncertainties  the  decision  maker  encountered.  The  DMC  focuses  on  the  cognitive 
aspects  of  decision  making  rather  than  the  procedural  or  tactical  aspects  that  are  more  likely  to 
be  addressed  in  traditional  AARs.  The  DMC  utilizes  the  following  questions  to  initiate 
discussion  pertaining  to  decisions  that  were  made  in  response  to  the  DMG  dilemma: 

•  What  were  the  tough  decisions? 

•  For  each  decision: 

-  Why  was  it  difficult? 

-  Why  did  you  choose  that  course  of  action? 

-  What  one  piece  of  missing  information  would  have  helped  you  most? 

-  What  other  actions  did  you  consider?  Why  didn’t  you  choose  them? 

At  the  close  of  the  session,  the  DMC  uses  the  following  general  discussion  questions  to 
summarize  the  lesson: 

•  What  would  you  do  differently  if  you  were  in  this  situation  again? 

•  What  was  our  vmit’s  biggest  strength?  Our  biggest  weakness? 

•  What  are  some  important  lessons  learned  from  this  exercise? 

A  program  of  DST  was  developed  for  platoon  leaders  to  improve  their  preparedness  in 
MOOT  environments.  MOUT  places  leaders  in  highly  demanding  situations  where  they  have  to 
operate  with  a  high  degree  of  independence.  Relatively  junior  officers  face  high  risks  and 
demanding  situations  with  very  little  preparation  for  handling  the  decisions  and  judgments  in  the 
MOUT  mission.  Given  the  Army’s  expectation  that  more  and  more  infantrymen  will  participate 
in  MOUT,  it  has  become  necessary  to  initiate  decision-centered  training  programs.  The  best 
candidates  to  receive  the  training  are  the  least  experienced — ^the  lieutenants. 

The  DST  program  incorporated  in  IMPACT  focuses  on  platoon  leader  decision  making  in 
MOUT  building  clearing  operations.  The  first  step  in  the  development  of  this  program  was  to 
conduct  a  Cognitive  Task  Analysis  with  MOUT  subject-matter  experts  (SMEs).  These  data  were 
analyzed  to  determine  the  most  critical  decision  requirements  in  the  domain  (Phillips  et  al.,  2001; 
Phillips,  McDermott,  Thordsen,  McCloskey,  &  Klein,  1998) 

•  Securing  the  perimeter 

•  Approaching  the  building 

•  Entering  the  building 

•  Clearing  the  building 

•  Maintaining  and  extending  security 

•  Evacuating  the  building 

•  Thinking  like  the  enemy 
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•  Maintaining  the  big  picture  and  situation  awareness 

•  Projecting  into  the  future 

•  Leading  subordinates 

•  Understanding  and  applying  Rules  of  Engagement 

Based  on  these  requirements  and  their  associated  Cognitive  demands,  IMPACT  was  tailored  for 
Army  instmctors  who  train  platoon  leaders  in  MOUT. 

Our  past  experience  conducting  DST  workshops  indicated  that  the  DMC  was  a  more 
effective  learning  tool  if  it  was  used  by  a  skilled  facilitator.  Therefore,  we  have  developed 
training  for  facilitation  skills.  We  also  learned  that  facilitators  like  to  develop  their  own  DMGs, 
so  we  have  developed  strategies  for  training  them  to  create  DMGs  for  particular  problems  faced 
by  their  students.  IMPACT  incorporates  the  following  five  modules: 

1)  An  introduction  to  DST 

2)  How  to  facilitate  Decision  Making  Games 

3)  How  to  employ  the  Decision  Making  Critique 

4)  How  to  create  Decision  Making  Game  scenarios 

5)  A  library  of  Decision  Making  Games  for  platoon  leaders  in  MOUT. 

Study  Overview 


Conducting  evaluations  of  interventions  in  real-world  settings  has  proved  to  be  a 
challenge  to  the  Naturalistic  Decision  Making  (NDM)  community.  Issues  such  as  how  to  assess 
the  impact  of  a  specific  intervention,  how  to  find  professionals  who  are  willing  and  able  to  take 
time  to  participate  in  a  study,  which  variables  to  control  to  increase  the  likelihood  of 
demonstrating  an  effect  without  creating  so  much  artificiality  that  the  evaluation  has  little 
meaning  in  the  real  world,  and  developing  appropriate  measures  to  assess  outcomes  all  present 
challenges.  This  paper  describes  an  effort  to  better  understand  these  methodological  issues  in  the 
context  of  a  preliminary  evaluation  of  a  multimedia  CD-ROM  based  training  program  that 
teaches  Army  leaders  how  to  develop  their  subordinates’  decision-making  skills. 

The  evaluation  was  designed  aroimd  two  questions.  First,  we  wanted  to  know  whether 
Army  officers  found  IMPACT  to  be  a  usable  tool.  We  were  particularly  interested  in  whether 
they  would  be  able  to  use  the  DMC  to  facilitate  DMG  sessions  based  on  the  training  they 
received  fi'om  IMPACT  alone  (without  participating  in  a  DST  workshop).  Second,  we  wanted  to 
know  whether  we  could  demonstrate  a  measurable  difference  in  the  decision-making 
performance  of  individuals  who  participated  in  DMG  sessions  led  by  an  IMPACT-trained 
facilitator  (the  IMPACT  groups)  versus  individuals  who  participated  in  DMG  sessions  led  by 
facilitators  who  had  not  had  access  to  IMPACT  (the  NO-IMPACT  groups).  If  we  had 
successfully  implemented  the  essence  of  our  DST  program  in  IMPACT,  then  we  should  see 
improved  decision-making  performance  fi’om  individuals  who  participated  in  DMG  sessions  led 
by  facilitators  who  used  IMPACT. 

The  study  design  is  detailed  in  the  following  sections.  However,  we  provide  an  overview 
here  to  aid  in  readability  of  the  report.  Cadets  in  their  junior  year  at  USMA  West  Point 


3 


participated  in  a  series  of  three  55-mmute  sessions  on  consecutive  days.  The  first  two  days  were 
training  sessions  in  which  students  participated  in  facilitator-led  DMG  sessions.  On  the  third  day, 
Klein  Associates  personnel  hosted  a  session  in  which  students  completed  the  dependent 
measures  for  the  study  (i.e.,  responded  to  a  test  DMG  and  filled  out  a  set  of  questionnaires). 

Half  of  the  students  were  assigned  to  the  IMPACT  group  and  participated  in  sessions  led  by 
IMPACT-trained  facilitators  for  their  training  sessions.  The  other  half  of  the  students  were  in 
the  NO-IMPACT  group  and  participated  in  sessions  led  by  facilitators  who  had  not  been  exposed 
to  IMPACT  and  who  were  instructed  to  use  a  standard  AAR  format  for  the  training  sessions. 

METHOD 

Participants 


Facilitators 

The  facilitators  were  recruited  from  a  pool  of  U.S.  Army  captains  currently  enrolled  in 
the  Tactical  Officer  Education  Program  (TOEP)  at  the  USMA  at  West  Point.  All  facilitators  were 
males  aged  30-33  years.  They  were  informed  by  their  commanding  officer  of  the  opportunity  to 
participate  in  a  study  evaluating  decision  training  for  cadets.  Sixteen  officers  expressed  their 
willingness  to  participate  by  completing  a  background  questionnaire  that  included  basic 
demographic  items  and  questions  about  their  prior  military  experience  and  training.  This 
information  was  used  to  match  12  of  the  volunteers  into  six  pairs  for  subsequent  random 
assignment  to  experimental  conditions.  The  criteria  for  matching  the  instructors  included  the 
amoimt  of  formal  instruction  officers  had  on  how  to  facilitate  group  feedback  sessions,  MOUT 
experience,  prior  years  experience  as  an  enlisted  soldier,  and  whether  they  graduated  from 
USMA.  Officers  were  offered  a  complimentary  copy  of  the  IMPACT  CD  for  their  participation. 

Cadets 


Sixty-one  male  and  14  female  cadets  were  recruited  from  junior-level  students  currently 
enrolled  in  the  Military  Leadership  or  Cognitive  Psychology  courses  in  the  Department  of 
Behavioral  Sciences  and  Leadership  at  USMA.  Data  from  eight  males  and  two  females  were  not 
included  in  subsequent  analyses  because  they  failed  to  complete  all  three  experimental  sessions. 
All  students  received  course  credit  for  their  participation. 

Materials 


Facilitator  Training  Packets 

All  packets  included  a  cover  letter  with  general  instructions,  paper  copies  of  the  DMGs, 
and  overhead  transparencies  of  the  corresponding  battlefield  scenarios.  The  DMGs  used  in  this 
evaluation  were  previously  rated  for  their  difficulty  level  as  described  in  Phillips  et  al.  (2001). 
Three  intermediate'  DMGs  were  selected  for  use  in  the  evaluation  study.  “Nightmare  on  Elm 


'  DMGs  of  intermediate  difficulty  were  chosen  in  order  to  ensure  that  the  scenarios  were  challenging  enough  to 
allow  for  variability  between  the  groups  based  on  their  learning  in  the  training  sessions,  but  not  so  challenging  that 
the  cadets/facilitators  with  limited  MOUT  experience  would  be  unable  to  effectively  use  the  DMGs. 
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Street”  was  used  on  Day  1,  “Caught  in  the  Intersection”  on  Day  2  and  “A  Deadly  Approach”  on 
Day  3.  The  DMGs  can  be  found  in  Appendix  A. 

Facilitators  in  the  IMPACT  group  received  a  beta  copy  of  IMPACT.  Facilitators  were 
instructed  to  only  look  at  three  of  the  five  modules  on  the  CD:  the  Introduction,  Facilitating 
DMGs,  and  the  Decision  Making  Critique  (DMC)  because  the  other  two  components.  Creating 
DMGs  and  the  DMG  Library  were  not  fully  functional. 

Both  groups  of  facilitators  received  instructions  on  “How  to  Conduct  a  Decision  Making 
Game”  that  included  step-by-step  guidance  for  conducting  the  DMG  sessions.  Copies  of  the 
instructions  for  both  groups  are  included  in  Appendix  B.  The  guidance  for  facilitators  in  the 
IMPACT  group  instructed  them  to  use  the  DMC  while  leading  group  discussion  of  the  DMG  and 
to  have  the  cadets  engage  in  a  role-play  as  part  of  the  discussion  of  the  DMG.  For  the  role-play, 
one  cadet  was  to  be  selected  from  Ae  class  to  play  the  role  of  the  platoon  leader  and  share  his/her 
course  of  action  by  issuing  orders  to  other  classmates  who  would  play  the  roles  of  the  squad 
leaders. 


The  guidance  for  the  facilitators  in  the  NO-IMPACT  group  instructed  them  to  use  a 
standard  A(^  format  while  leading  the  group  discussion.  They  were  not  told  to  have  the  cadets 
engage  in  a  role-play  as  part  of  the  discussion  of  the  DMG.  Facilitators  in  the  NO-IMPACT 
group  also  received  a  handout  called,  “A  Leader’s  Guide  to  After  Action  Reviews,”  which  was 
based  on  information  from  the  Center  for  Army  Lessons  Learned  (CALL)  website 
(httD://call.armv.mil/call/spc  prod/tc25-20/ guide.htm). 

Dependent  Measures 

There  were  four  basic  types  of  dependent  measures  used  to  examine  group  differences  in 
the  evaluation  study:  cadets’  written  responses  to  the  DMGs,  cadets’  responses  to  a  multiple- 
choice  test  assessing  their  knowledge  of  MOUT,  cadets’  responses  to  questionnaires  that  elicited 
their  subjective  evaluation  of  the  decision  training  sessions,  and  the  facilitators’  responses  to 
questionnaires  that  elicited  their  subjective  evaluation  of  the  decision  training  sessions.  In 
addition,  facilitators  in  the  IMPACT  group  were  also  asked  to  provide  feedback  on  the  IMPACT 
CD. 


Cadet  DMG  responses.  Cadets  were  required  to  write  down  their  responses  to  the  DMGs 
on  response  forms  during  both  the  training  sessions  (Day  1  and  Day  2)  and  the  test  session  (Day 
3).  The  DMG  Response  Form  used  on  Days  1  and  2  was  a  one-page  form  that  asked  the  cadets  to 
record  their  proposed  course  of  action  in  the  form  of  orders  to  be  given  to  their  squad  leaders  and 
other  units  if  applicable.  The  response  form  also  asked  the  cadets  to  describe  their  assessment  of 
the  current  situation.  The  cadets  were  given  four  minutes  to  complete  this  form.  The  DMG 
Response  Form  used  on  Day  3  was  a  two-page  form  that  was  completed  in  two  stages.  First  the 
cadets  were  asked  to  complete  the  first  page  which  asked  them  to  record  their  proposed  course  of 
action  in  the  form  of  orders  to  be  given  to  their  squad  leaders  and  other  units  if  applicable  (as 
they  had  been  asked  to  do  on  Days  1  and  2).  The  cadets  were  given  three  minutes  to  complete 
page  one  of  the  DMG  Response  Form.  The  second  page  required  the  cadets  to  describe  their 
assessment  of  the  situation,  to  explain  what  they  were  trying  to  accomplish  with  their  plan  and  to 


5 


describe  their  greatest  concern  given  the  situation  described  in  the  DMG  scenario.  Cadets  were 
given  an  additional  five  minutes  to  complete  the  second  page  of  the  DMG  response  form.  The 
expanded  version  of  the  DMG  Response  Form  was  used  on  Day  3  in  order  to  elicit  a  more 
detailed  rationale  of  the  cadet’s  proposed  course  of  action  for  subsequent  evaluation  by  SMEs. 
Copies  of  both  response  forms  are  included  in  Appendix  C. 

The  quality  of  cadets’  responses  to  the  DMG  presented  during  the  test  session  was  rated 
by  independent  SMEs  who  were  blind  to  condition.  Two  SMEs  with  extensive  MOUT 
experience  were  selected  to  rate  cadet  responses  to  the  DMG.  The  SME  ratings  were  conducted 
after  all  the  data  had  been  collected.  In  a  previous  DST  evaluation  study  (McCloskey,  Pliske, 
Klein,  Heaton,  &  Knight,  1997)  we  had  difficulty  establishing  acceptable  inter-rater  reliability 
between  the  SMEs  who  evaluated  the  participants’  DMGs  responses.  Based  on  the  lessons 
learned  from  our  previous  effort,  we  developed  the  following  procedure  in  order  to  ensure  we 
had  reliable  estimates  of  the  quality  of  the  cadets’  DMG  responses. 

In  order  to  familiarize  the  SMEs  with  the  DMG,  they  participated  in  a  mock  DMG 
session  facilitated  by  a  member  of  the  Klein  Associates’  research  team.  After  each  SME  had 
presented  his  COA,  the  facilitator  led  a  discussion  of  the  critical  pieces  of  information  included 
in  the  DMG  scenario.  The  facilitator  then  reviewed  the  set  of  seven  rating  scales  that  the  SMEs 
used  to  independently  evaluate  the  cadets’  DMG  responses.  The  rating  scales  are  shown  in  Table 
1.  Each  was  a  5 -point  Likert-type  scale,  where  a  rating  of  5  indicated  “strongly  agree,”  a  rating 
of  1  indicated  “strongly  disagree,”  and  a  rating  of  3  indicated  “neither  agree  nor  disagree.” 

Table  1 


Description  of  Scales  Used  by  SMEs  to  Evaluate  Cadets’  (N  =  65f  Written  Responses  to  the  Test 
DMG  and  Inter-Rater  Reliabilities 


Scale  description 

Reliability 

1 .  This  plan  is  acceptable,  given  the  situation. 

.92 

2.  This  plan  is  good,  given  the  mission. 

.93 

3.  This  plan  is  very  similar  to  the  course  of  action  I  would  have  carried  out 
in  this  situation. 

.93 

4.  The  orders  issued  provide  the  squad  leaders  with  a  clear  idea  of  what 
they  are  supposed  to  accomplish. 

.82 

5.  The  assessment  of  the  situation  indicates  the  participant  was  paying 
attention  to  the  most  important  information. 

.84 

6.  The  elaboration  of  the  plan  indicates  that  reasoning  behind  the  proposed 
course  of  action  was  soimd. 

.89 

7.  The  greatest  concern  reported  by  the  participant  is  an  important  concern 
in  this  type  of  battle  situation. 

.89 
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The  first  three  ratings  scales  elicit  SME  judgments  about  the  overall  goodness  of  the 
cadet’s  proposed  COA.  Because  the  DMG  used  in  the  test  session  has  many  different  possible 
solutions,  we  wanted  to  first  establish  the  acceptability  of  the  plan  and  then  have  the  SME  make 
the  more  subjective  evaluation  of  the  “goodness”  of  the  plan.  We  included  the  third  scale  to 
allow  the  SMEs  the  opportunity  to  indicate  that  a  specific  plan  was  very  different  fi'om  their  own 
solution.  We  thought  this  scale  would  help  remind  the  SMEs  that  a  participant  could  propose 
an  acceptable  COA  that  was  different  from  the  COA  the  SME  preferred.  The  fourth  rating  scale 
asked  the  SMEs  to  judge  the  clarity  of  the  orders  described  by  the  cadet.  The  fifth  rating  scale 
assessed  the  quality  of  the  cadet’s  situation  assessment.  The  sixth  scale  was  included  to  evaluate 
the  quality  of  the  rationale  provided  by  the  cadet  for  the  particular  COA  he/she  developed.  The 
last  scale  asked  the  SME  to  evaluate  whether  the  cadet’s  greatest  worry  was  an  important 
concern  in  this  MOUT  situation. 

The  SMEs  reviewed  a  DMG  response  from  a  cadet  who  participated  in  the  test  session, 
but  failed  to  show  up  for  both  training  sessions  and  was  therefore  not  included  in  the  analyses. 
Our  researcher  facilitated  the  SMEs’  discussion  of  how  they  would  rate  this  particular  response 
in  order  to  ensure  they  were  interpreting  the  scales  correctly.  The  SMEs  then  proceeded  to  rate 
the  DMG  responses  independently.  After  completing  a  set  of  six  DMG  response  forms,  our 
researcher  facilitated  a  discussion  between  the  SMEs  in  to  order  reach  a  consensus  for  any  scale 
on  to  which  they  disagreed  by  two  or  more  points  on  the  rating  scale.  The  SMEs  continued  to 
make  their  ratings  independently  imtil  all  of  the  cadets’  DMG  responses  had  been  evaluated.  At 
times,  the  SMEs  helped  each  other  decipher  cadets’  handwritten  responses  on  the  DMG  response 
forms.  The  Klein  Associates’  researcher  reviewed  the  SMEs’  independent  evaluations  to  identify 
cases  where  the  ratings  differed  by  two  or  more  scale  points.  She  had  the  SMEs  discuss  these 
DMGs  and  reach  consensus.  The  SMEs  had  to  discuss  9  of  the  455  ratings  (2%)  on  the  DMG 
response  forms  in  order  to  resolve  discrepancies.  As  shown  in  Table  1,  the  inter-rater  reliability 
for  their  final  ratings  using  the  seven  scales  was  extremely  high  (.82  -  .93).  The  average  rating 
given  by  the  two  SMEs  was  used  in  subsequent  analyses. 

Cadet  MOUT  knowledge  test.  A  MOUT  Knowledge  Test  was  developed  with  the  help  of 
the  two  SMEs  with  extensive  MOUT  experience.  (These  were  not  the  same  SMEs  who  rated 
cadet  responses  to  the  DMG.)  It  consisted  of  two  brief  MOUT  scenarios  followed  by  eight 
multiple-choice  questions  per  scenario  for  a  total  of  16  questions.  A  scoring  key  for  this  test  was 
developed  based  on  answers  provided  by  the  two  SMEs  who  rated  cadet  responses,  and  one  SME 
who  was  involved  in  the  development  of  the  MOUT  Knowledge  Test.  For  nine  items  the  three 
SMEs  agreed  that  one  response  was  correct.  However,  for  seven  items,  there  was  a  lack  of 
consensus  among  the  three  SMEs,  so  multiple  answers  were  counted  as  correct  when  scoring  the 
cadets’  responses.  Specifically,  a  cadet’s  response  was  counted  as  being  “correct”  as  long  as  at 
least  one  of  the  SMEs  had  selected  that  answer.  A  copy  of  the  MOUT  questiormaire  that  includes 
the  scoring  key  is  shown  in  Appendix  D. 

Cadet  questionnaires.  Cadets  were  asked  to  complete  brief  questionnaires,  called  Session 
Evaluation  Forms,  at  the  end  of  the  Day  1  and  Day  2  training  sessions.  They  completed  a  similar 
form  (the  Cadet  Post-Test  Questiormaire)  as  part  of  the  test  session  on  Day  3.  They  were  asked  to 
indicate  the  extent  to  which  they  agreed  or  disagreed  with  ten  statements  by  circling  a  number  on 
a  5-point  scale  where  1  was  labeled  “strongly  disagree”  and  5  was  labeled  “strongly  agree.” 
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These  statements  assessed  whether  or  not  the  participants  believed  the  DMG  sessions  had 
improved  their  ability  to  make  rapid  decisions  on  the  battlefield  and  their  assessment  of  the 
effectiveness  of  the  facilitator  who  led  the  DMG  sessions.  Copies  of  these  questionnaires  are 
included  in  Appendix  E. 

Facilitator  questionnaires.  Facilitators  were  asked  to  complete  a  brief  questionnaire, 
called  the  Facilitator  Post-Session  Questionnaire,  after  they  completed  the  Day  2  training 
session.  Both  groups  of  facilitators  were  asked  to  indicate  the  extent  to  which  they  agreed  or 
disagreed  with  1 1  statements  by  circling  a  number  on  a  5-pomt  scale  where  1  was  labeled 
“strongly  disagree”  and  5  was  labeled  “strongly  agree.”  These  statements  assessed  whether  or 
not  the  facilitators  believed  the  DMG  sessions  had  improved  the  cadets’  ability  to  make  rapid 
decisions  on  the  battlefield  and  the  facilitators’  self-assessment  of  their  effectiveness  in  leading 
the  DMG  sessions.  Additional  questions  on  the  questionnaire  elicited  information  about  how 
much  time  the  facilitators  had  spent  in  preparation  for  the  training  sessions,  their  suggestions  for 
improving  the  DMGs  used  in  the  training  sessions,  and  what  types  of  additional  information  they 
would  have  liked  to  have  had  prior  to  the  training  sessions.  In  addition,  facilitators  in  the 
IMPACT  group  were  asked  to  evaluate  the  different  IMPACT  training  modules  and  provide  their 
subjective  assessment  of  the  IMPACT  CD.  Copies  of  these  questionnaires  are  included  in 
Appendix  F. 

Procedure 


A  matched-groups  design  was  used  to  evaluate  the  effectiveness  of  IMPACT-trained 
facilitators.  After  matching  the  12  facilitators  based  on  their  previous  military  experience  and 
training,  one  facilitator  from  each  of  the  6  pairs  was  randomly  assigned  to  the  IMPACT  or  NO¬ 
IMPACT  condition.  Each  facilitator  was  required  to  present  a  DMG  and  lead  the  discussion  for 
the  same  group  of  3-8  cadets  on  two  consecutive  days.  There  were  a  total  of  six  IMPACT  groups 
and  six  NO-IMPACT  groups.  Cadets  were  randomly  assigned  to  participate  in  either  an 
IMPACT  or  NO-IMPACT  group. 

Data  collection  took  place  over  a  two- week  period.  Dining  each  week,  data  from  three 
IMPACT  groups  and  three  NO-IMPACT  groups  were  collected.  Data  collection  involved  three 
55-minute  sessions  for  each  group  of  cadets  on  three  consecutive  days.  On  Days  1  and  2,  cadets 
met  with  their  assigned  facilitator,  participated  in  a  DMG,  and  completed  the  DMG  Session 
Evaluation  Form.  On  Day  3,  the  cadets  met  with  a  Klein  Associates’  researcher  who 
administered  a  test  DMG,  the  MOUT  Knowledge  Test,  and  the  Cadet  Post-Test  Questionnaire. 

Facilitators  received  an  instmctional  packet  five  days  prior  to  their  first  DMG  session. 
Facilitators  in  the  IMPACT  group  were  sent  a  beta  copy  of  the  IMPACT  CD  and  asked  to  spend 
2-3  hours  reviewing  the  information  provided  on  the  CD  prior  to  their  first  DMG  session. 
Facilitators  in  both  groups  were  asked  to  review  the  DMG  materials  prior  to  their  training 
sessions.  For  each  week,  prior  to  the  Day  1  session,  facilitators  met  as  a  group  with  Klein 
Associates’  personnel  to  ask  any  last  minute  questions  on  logistics,  procedures,  and  so  forth. 
Facilitators  were  given  their  room  assignments  and  the  names  of  the  cadets  who  would  be 
participating  in  their  groups. 
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At  the  beginning  of  the  Day  1  session,  the  facilitators  gave  a  brief  overview  to  the  group 
of  cadets  on  what  they  would  be  asked  to  do  over  the  course  of  the  evaluation  study.  In  all  other 
respects  the  same  procedures  were  followed  for  Days  1  and  2.  Facilitators  passed  out  the  DMG 
response  forms  for  the  cadets  to  record  their  answers  to  the  DMG  and  stressed  the  importance 
of  writing  neatly.  Facilitators  then  read  the  DMG  scenario  and  allowed  cadets  to  ask  questions 
about  the  scenario.  Facilitators  were  encouraged  to  answer  questions  that  would  clarify  the 
scenario  for  the  cadets.  For  example,  if  a  cadet  asked  a  question  that  suggested  he/she  had  failed 
to  hear  some  of  the  information  provided  in  the  scenario,  then  the  facilitator  should  repeat  that 
information.  Facilitators  were  cautioned  that  if  they  chose  to  answer  questions  that  involved 
supplementing  the  information  specifically  provided  in  the  written  scenario,  their  responses 
needed  to  be  consistent  with  the  information  provided  in  the  scenario. 

The  facilitators  were  instructed  to  give  the  cadets  four  minutes  to  record  their  courses 
of  action  on  the  response  forms  in  the  form  of  orders  to  be  given  to  their  squad  leaders.  The 
response  form  also  asked  the  cadets  to  describe  their  assessment  of  the  current  situation.  After 
the  four-minute  time  period  expired,  an  individual  cadet  was  asked  to  present  his/her  course  of 
action  and  the  facilitator  led  the  subsequent  discussion.  At  the  end  of  the  session,  the  facilitators 
passed  out  Session  Evaluation  Forms.  The  cadets’  responses  on  these  forms  were  anonymous 
and  the  facilitator  left  the  room  while  the  cadets  completed  the  evaluation.  Facilitators  were 
given  the  Facilitator  Post-Session  Questioimaire  to  complete  after  their  Day  2  session. 

On  Day  3,  the  cadets  met  with  Klein  Associates’  researchers  who  administered  a  set  of 
assessment  measures.  One  Klein  Associates’  researcher  met  with  cadets  in  the  IMPACT  groups 
and  another  researcher  met  with  the  cadets  in  the  NO-IMPACT  groups.  Different  Klein 
Associates  researchers  ran  the  test  sessions  during  weeks  one  and  two.  During  the  Day  3  test 
session,  the  researchers  presented  a  DMG  and  gave  participants  three  minutes  to  record  their 
course  of  action  on  their  response  form.  Cadets  were  then  given  five  additional  minutes  to 
describe  their  assessment  of  the  situation,  to  explain  what  they  were  trying  to  accomplish  with 
their  plan,  and  to  describe  their  greatest  concern  given  the  situation  described  in  the  DMG 
scenario.  No  group  discussion  of  the  DMG  took  place  during  the  Day  3  test  session.  After  the 
cadets  had  completed  the  DMG  response  forms,  the  research  team  administered  MOUT 
Knowledge  Tests,  followed  by  the  Cadet  Post-Test  Questionnaire.  Lastly,  a  very  generic  written 
debrief  about  the  evaluation  study  was  handed  out  and  cadets  were  invited  to  attend  an  optional 
Decision  Skills  Training  Night,  where  they  were  provided  with  a  more  detailed  debrief  about  the 
evaluation  study  and  participated  in  another  DMG  session  led  by  an  experienced  facilitator.  The 
Decision  Skills  Training  Night  took  place  following  the  completion  of  the  data  collection  during 
week  two  in  order  to  avoid  potential  “contamination”  of  the  week  two  participants.^ 


^  One  cadet  attended  the  Decision  Skills  Training  Night.  The  other  attendees  were  USMA  faculty  members  and 
captains  who  had  participated  in  the  study  as  facilitators. 
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RESULTS 


Cadet  DMG  Responses 

Table  2  summarizes  the  SMEs’  average  ratings  of  the  cadets’  written  responses  to  the  test 
DMG  administered  on  Day  3.  An  examination  of  the  means  for  the  two  different  conditions  for 
the  seven  rating  scales  indicates  that  the  DMG  responses  from  cadets  in  both  the  IMPACT  and 
NO-IMPACT  conditions  received  similar  ratings  from  the  SMEs.  The  average  rating  for  groups 
of  cadets  was  arotmd  “3”  on  a  5-point  scale  where  higher  numbers  indicate  a  more  positive 
evaluation  by  the  SMEs.  A  MANOVA  using  condition  and  testing  week  as  between-subject 
factors  showed  no  significant  effects  for  condition,  F(7, 55)  <  1.  Nor  was  there  a  significant 
effect  due  to  testing  week,  F  (7, 55)  =  1.61,  p  >  .05;  or  a  significant  interaction  between  week 
and  condition,  F  (7,55)  <  1. 

Table  2 

SMEs’  Average  Ratines  of  Cadet  Responses  to  the  Test  DMG  for  IMPACT  and  NO-IMPACT 
Conditions 


SME  Rating  Scale 

Condition 

M* 

SD 

1 .  This  plan  is  acceptable,  given  the  situation. 

IMPACT 

3.03 

1.39 

NO-IMPACT 

3.21 

1.34 

2.  This  plan  is  good,  given  the  mission. 

IMPACT 

2.76 

1.33 

NO-IMPACT 

2.90 

1.30 

3.  This  plan  is  very  similar  to  the  course  of  action  I 

IMPACT 

2.25 

1.42 

would  have  carried  out  in  this  situation. 

NO-IMPACT 

2.45 

1.52 

4.  The  orders  issued  provide  the  squad  leaders  with  a 

IMPACT 

338 

1.00 

clear  idea  of  what  they  are  supposed  to  accomplish. 

3.39 

5.  The  assessment  of  the  situation  indicates  the 

IMPACT 

3.25 

1.12 

participant  was  paying  attention  to  the  most 

important  information. 

NO-IMPACT 

3.29 

.91 

6.  The  elaboration  of  the  plan  indicates  that  reasoning 

IMPACT 

3.13 

1.26 

behind  the  proposed  course  of  action  was  sound. 

NO-IMPACT 

3.13 

1.07 

7.  The  greatest  concern  reported  by  the  participant  is  an 

IMPACT 

3.06 

1.23 

important  concern  in  this  type  of  battle  situation. 

NO-IMPACT 

3.35 

.94 

*  SMEs  used  a  5-point  rating  scale  where  1  indicated  “strongly  disagree”  and  5  indicated 
“strongly  agree.” 
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Cadet  MOUT  Knowledge  Test 


The  mean  proportion  of  correct  responses  to  the  multiple-choice  MOUT  Knowledge 
Test  was  computed  for  each  cadet  in  the  IMPACT  and  NO-IMPACT  conditions.  Although  the 
difference  between  the  conditions  was  in  the  predicted  direction  with  the  cadets  in  the  IMPACT 
groups  having  a  higher  mean  proportion  of  correct  responses  (M  =  .67,  SD  =  .10)  than  the  NO¬ 
IMPACT  cadets  (M  =  .64,  SD  =  .1 1),  a  two  factor  ANOVA  indicated  that  this  result  is  not 
statistically  reliable  [F  (1, 61)  =  1.77,  p  >  .05].  The  ANOVA  also  indicated  there  was  no  reliable 
difference  between  the  mean  proportion  of  correct  responses  for  the  different  testing  weeks, 
F(1,61)<1;  nor  was  there  a  significant  interaction  between  testing  week  and  condition, 
F(l,61)=1.37,p>.05. 

Cadet  Questionnaires 

The  means  and  standard  deviations  for  cadets’  responses  on  the  DMG  Session  Evaluation 
Forms  are  shown  in  Table  3  for  the  IMPACT  and  NO-IMPACT  conditions  for  each  of  the 
10  rating  scales.  These  data  indicate  that  overall  the  cadets  in  both  conditions  rated  the  DMG 
sessions  positively  on  these  5-point  scales  where  high  numbers  indicate  positive  evaluations.  In 
general,  cadets  in  the  IMPACT  condition  rated  the  DMG  sessions  somewhat  more  positively 
than  the  cadets  in  the  NO-IMPACT  condition,  however  this  trend  does  not  hold  for  both  weeks 
and  both  training  days. 

In  order  to  get  a  better  understanding  of  these  data,  an  average  evaluation  rating  was 
computed  for  each  cadet  by  summing  his  or  her  ratings  on  all  of  the  scales  except  question 
number  8  which  asked  the  cadets  whether  or  not  the  MOUT  scenario  used  in  the  DMG  session 
was  challenging.  We  had  included  item  8  on  the  DMG  Session  Evaluation  Form  to  determine 
whether  or  not  the  cadets  in  the  IMPACT  groups  differed  from  the  cadets  in  the  NO-IMPACT 
groups  in  terms  of  their  level  of  sophistication  about  MOUT. 

A  mixed  ANOVA  with  condition  and  group  as  between-subject  factors  and  day  as  a 
within-subject  factor  indicated  there  is  a  significant  main  effect  of  day  F  (1, 61)  =  31.23,  p  <  .05, 
and  a  significant  three-way  interaction  between  condition,  week,  and  day,  F(l,  61)  =  5.68,  p< 

.05.  There  were  no  significant  main  effects  of  week,  F  (1,61)  =  2.28,  p>  .05,  or  condition  F  <  1. 
Nor  was  there  a  significant  interaction  between  week  and  day,  F  (1,61)  =  2.22,  p  >  .05,  or  week 
and  condition,  F  <  1 .  An  examination  of  the  means  indicated  that  cadets  reported  the  DMG  was 
more  challenging  given  their  current  knowledge  of  MOUT  on  Day  2  (M  =  4.27,  SD  =  85)  than 
on  Day  1  (M  =  3.66,  SD  =1.13).  Furthermore,  although  the  cadets  in  the  IMPACT  condition 
typically  reported  slightly  higher  nrunbers  for  this  scale,  on  the  first  day  of  training  during  the 
second  week  of  testing,  cadets  in  the  NO-IMPACT  condition  reported  higher  numbers  (M  = 
3.58,  SD  =  1.07)  as  compared  to  cadets  in  the  IMPACT  condition  (M  =  3.27,  SD  =  1.42). 

We  had  predicted  that  cadets  in  the  IMPACT  condition  would  have  higher  ratings  than 
cadets  in  the  NO-IMPACT  condition  for  the  other  nine  items  on  the  DMG  Session  Evaluation 
Form.  The  summarized  ratings  were  analyzed  with  a  mbced  ANOVA  with  condition  and  group 
as  between-subject  factors  and  day  as  a  within-subject  factor.  This  analysis  indicates  there  is  a 
significant  three-way  interaction  between  condition,  week,  and  day,  F(l,  61)  =  6.20,  p<  .05. 
There  was  also  a  significant  main  effect  of  day,  F  (1, 61)=14.64,  p<  .05,  a  significant  main  effect 
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of  week,  F  (1,61)  =2.51,  p<  .05,  and  a  significant  interaction  between  week  and  day,  F  (1,61)  = 
16.02.  The  main  effect  of  condition  was  not  significant,  F(1,61)<1,  nor  was  the  interaction 
between  week  and  condition  significant,  F  (1,61)  <  1. 

Table  3 


Cadet  Responses  to  the  DMG  Session  Evaluation  Form  for  the  IMPACT  and  NO-IMPACT 
Conditions 


Rating  Scale 

Condition 

M* 

SD 

1 .  T oday ’  s  DMG  session  was  a  valuable  learning 

IMPACT 

4.25 

.64 

experience  for  me. 

m 

NO-IMPACT 

4.19 

2.  Participation  in  today’s  DMG  session  should 

IMPACT 

4.17 

.65 

improve  my  ability  to  make  rapid  decisions  on  the 

battlefield. 

NO-IMPACT 

4.11 

.75 

3.  I  learned  a  great  deal  of  new  information  about 

IMPACT 

3.41 

.89 

MOUT  in  today’s  DMG  session. 

NO-IMPACT 

3.45 

.84 

4.  The  facilitator  did  an  outstanding  job  of  leading 

IMPACT 

4.48 

.60 

today’s  DMG  session. 

NO-IMPACT 

4.26 

.68 

5.  Today’s  DMG  session  provided  participants  with  a 

IMPACT 

4.15 

.66 

valuable  opportunity  to  practice  their 

.64 

communications  skills. 

NO-IMPACT 

3.90 

6.  Today’s  DMG  session  provided  participants  with  a 

IMPACT 

4.54 

.50 

valuable  opportunity  to  practice  their  decision¬ 

making  skills. 

NO-IMPACT 

4.21 

.73 

7.  The  facilitator  helped  me  think  about  the  many 

IMPACT 

4.38 

.58 

different  factors  involved  in  making  rapid 

decisions  in  MOUT  settings 

NO-IMPACT 

4.24 

.72 

8.  The  MOUT  scenario  described  in  the  DMG  was 

IMPACT 

4.04 

.92 

challenging  given  my  current  knowledge  of 

.86 

MOUT. 

NO-IMPACT 

3.89 

9.  The  facilitator  kept  all  of  the  students  actively 

IMPACT 

4.48 

.51 

involved  during  the  DMG  session. 

NO-IMPACT 

4.35 

.76 

10. 1  would  like  to  participate  in  more  DMG  sessions 

IMPACT 

432 

.76 

in  the  future. 

NO-IMPACT 

4.11 

.87 

Cadets  utilized  a  5 -point  rating  scale  where  1  indicated  “strongly  disagree”  and  5  indicated 
“strongly  agree.” 


In  general,  cadets  rated  the  Day  2  sessions  (M  =  4.25,  SD  =  .56)  higher  than  the  Day  1 
sessions  (M  =  4.09,  SD  =  .45).  The  difference  between  the  Day  1  and  Day  2  sessions  was  greater 
during  Week  1  than  during  Week  2.  Overall,  the  ratings  for  the  IMPACT  conditions  were  higher 
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than  the  NO-IMPACT  condition,  except  on  Day  2  during  Week  1 .  These  differences  are 
summarized  in  Figure  1. 


D  a  y  1 
Sum 


D ay  2 
Sum 

”  Bars  show  Means 


Week 


C  o  n  d  itio  n 
I.  -J  Im  pact 
I  I  No  Im  pact 


Figure  1.  Summary  of  cadet  ratings  from  the  DMG  Session  Evaluation  Form  for  IMPACT 
and  NO-IMPACT  groups  for  Day  1  and  Day  2  for  Week  1  and  Week  2.  The  graphs  indicate 
that  the  IMPACT  groups  had  more  positive  evaluations  than  the  NO-IMPACT  groups  on 
every  day  except  Day  2,  Week  1 . 
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In  order  to  determine  if  the  observed  differences  on  the  DMG  Session  Evaluation  Form 
were  due  to  the  facilitators,  rather  than  condition,  the  summarized  ratings  were  subjected  to  a 
mixed  ANOVA  with  facilitators  as  between-subject  factors  and  day  as  a  within-subject  factor. 

As  in  the  above  analysis  there  was  a  significant  main  effect  of  day,  F  (1,  53)  =  14.83,  p  <  .05. 
There  was  also  a  marginally  significant  main  effect  for  facilitator,  F  (1 1,  53)  =  1 .97,  p  <  .06;  and 
a  significant  day  by  facilitator  interaction,  F  (1 1,  53)  =  2.75. 

The  means  and  standard  deviations  for  the  summary  ratings  for  the  different  facilitators 
are  shown  in  Table  4.  An  examination  of  these  means  indicates  that  of  the  two  facilitators  with 
the  highest  summary  ratings  (M  =  4.79  and  M  =  4.68),  one  was  in  the  IMPACT  condition  and 
one  was  in  the  NO-IMPACT  condition.  Likewise,  of  the  two  facilitators  with  the  lowest 
summary  ratings  (M  =  3.72  and  M  =  3.72),  one  was  in  the  IMPACT  condition  and  one  was  in  the 
NO-IMPACT  condition.  Therefore,  it  appears  that  although  there  were  reliable  differences 
between  the  facilitators,  these  differences  were  not  confoimded  with  condition. 

Table  4 


Cadets’  Summary  Ratings  on  the  DMG  Session  Evaluation  Form  for  Each  Facilitator 


Condition 

Week 

Facilitator 

M 

SD 

N 

IMPACT 

1 

A 

4.14 

.57 

6 

1 

B 

4.15 

.51 

5 

1 

C 

4.79 

.28 

5 

2 

D 

3.72 

.53 

4 

2 

E 

4.13 

.35 

6 

2 

F 

4.38 

.34 

8 

NO-IMPACT 

1 

G 

4.17 

.44 

3 

1 

H 

4.68 

.25 

4 

1 

I 

4.12 

.30 

5 

2 

J 

4.08 

.38 

6 

2 

K 

3.72 

1.09 

6 

2 

L 

4.17 

.35 

7 

Note.  The  facilitators  in  bold  type  had  the  highest  summary  ratings,  while  the  facilitators  in 
shaded  italics  had  the  lowest  summary  ratings. 

The  means  and  standard  deviations  for  cadets’  responses  on  the  Cadet  Post-Test 
Questionnaire  are  shown  in  Table  5  for  the  IMPACT  and  NO-IMPACT  conditions  for  each  of  10 
rating  scales.  These  data  indicate  that  overall  the  cadets  in  both  conditions  rated  the  DMG 
sessions  positively  on  the  5-point  scales  where  high  numbers  indicate  positive  evaluations.  In 
general,  cadets  in  the  IMPACT  condition  rated  the  DMG  sessions  somewhat  more  positively 
than  the  cadets  in  the  NO-IMPACT  condition,  however  this  effect  was  not  found  to  be 
significantly  reliable.  A  two-factor  ANOVA  with  condition  and  week  as  between-subject 
factors  was  conducted  on  the  average  evaliiation  rating,  which  was  computed  for  each  cadet 
by  summing  his  or  her  ratings  on  all  of  the  scales  except  question  number  8.  There  was  no 
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significant  main  effect  of  condition,  F  <  1 .  Nor  was  there  a  significant  main  effect  due  to  week 
F  <  1 ,  or  a  significant  interaction  between  condition  and  week,  F  <  1 . 

Table  5 

Cadet  Responses  to  the  Post-Test  Questionnaire  for  IMPACT  and  NO-IMPACT  Conditions 


Rating  Scale 

Condition 

M 

SD 

1 .  Participation  in  DMG  sessions  should  improve  my 

IMPACT 

.62 

ability  to  make  rapid  decisions  on  the  battlefield. 

HH 

NO-IMPACT 

.87 

2.  I  learned  a  great  deal  about  MOUT  by 

IMPACT 

1.08 

participating  in  the  DMG  sessions. 

NO-IMPACT 

3.65 

.88 

3.  I  learned  a  great  deal  of  new  information  about 

IMPACT 

4.18 

.76 

MOUT  in  today’s  DMG  session. 

NO-IMPACT 

4.16 

4.  DMG  sessions  provide  cadets  with  valuable 

IMPACT 

4.39 

opportunities  to  practice  their  decision-making 

■H 

skills. 

NO-MPACT 

4.32 

5 .  Participating  in  the  DMG  sessions  made  me  think 

IMPACT 

4.59 

.61 

about  Ae  many  different  factors  involved  in 

making  rapid  decisions  in  MOUT  settings. 

NO-IMPACT 

4.35 

.87 

6.  The  MOUT  scenarios  described  in  the  DMG 

IMPACT 

4.39 

sessions  were  challenging  given  my  current 

■H 

knowledge  of  MOUT. 

NO-IMPACT 

4.12 

7.  I  would  like  to  participate  in  DMG  sessions  in  the 

IMPACT 

4.23 

.83 

fixture. 

NO-IMPACT 

4.08 

1.04 

8.  DMG  sessions  demonstrated  that  other  cadets  can 

IMPACT 

4.41 

.76 

look  at  the  same  situation  and  come  up  with  very 

different  (but  reasonable)  courses  of  action. 

NO-IMPACT 

.92 

9.  DMG  sessions  demonstrated  that  the  decisions 

IMPACT 

4.43 

.69 

platoon  leaders  have  to  make  in  MOUT  situations 

involve  a  surprising  number  of  subtle  judgments. 

NO-MPACT 

4.23 

.79 

10.  Participation  in  the  DMG  sessions  should  improve 

IMPACT 

4.29 

.69 

my  ability  to  communicate  orders  to  my 

subordinates. 

NO-MPACT 

4.12 

.88 

Note.  No  significant  differences  were  found. 


Facilitator  Questionnaires 

The  officers  who  participated  in  both  the  IMPACT  and  NO-IMPACT  conditions  were 
asked  to  rate  1 1  items  that  elicited  their  self-assessment  of  their  skill  at  facilitating  the  DMG 
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sessions  and  the  usefulness  of  these  sessions.  The  means  and  standard  deviations  for  their 
responses  are  shown  in  Table  6.  A  set  of  t-tests  indicated  that  there  are  no  significant  differences 
between  the  IMPACT  and  NO-IMPACT  conditions  for  any  of  the  items. 

The  officers  who  participated  in  both  the  IMPACT  and  NO-IMPACT  conditions  were 
also  asked  a  series  of  open-ended  questions  to  determine  how  much  time  they  had  spent  in 
preparation  for  the  DMG  sessions,  what  else  we  could  have  done  to  better  prepare  them  for 
leading  these  sessions,  and  suggestions  for  how  to  improve  the  DMGs.  On  the  average,  officers 
in  the  IMPACT  condition  spent  130  minutes  (SD  =  30.98)  preparing  for  the  sessions  and  officers 
in  the  NO-IMPACT  condition  spent  55  minutes  (SD  =  34.35).  When  asked  what  we  could  have 
done  to  better  prepare  them  for  leading  the  DMG  sessions,  two  of  the  six  facilitators  in  each 
condition  replied  “nothing,”  whereas  all  of  the  other  officers  indicated  they  would  have  liked 
more  background  information  on  the  specific  scenario  or  more  information  on  MOOT.  The 
officers  in  both  conditions  seemed  to  like  the  DMGs  used  in  the  Day  1  and  Day  2  sessions, 
however,  several  of  them  had  suggestions  for  improving  them. 

The  officers  who  participated  in  the  IMPACT  condition  were  also  asked  to  evaluate  the 
usefulness  of  the  IMPACT  CD.  In  general,  they  had  very  favorable  responses  to  the  IMPACT 
CD.  The  average  “usefulness”  rating  of  the  different  instmctional  modules  was  a  “4”  on  a  5- 
point  scale  where  5  was  labeled  “extremely  useful”  and  1  was  labeled  “not  at  all  useful.”  The 
mean  rating  of  the  usefulness  of  the  Introduction  Module  was  4.5  (SD  =  .55);  the  mean  rating  of 
the  usefulness  of  the  module  on  how  to  facilitate  a  DMG  was  3.67  (SD  =  1.03);  and  the  mean 
rating  of  the  usefulness  of  the  DMC  module  was  4.33  (SD  =  .82). 

When  asked  what  they  didn’t  like  about  the  IMPACT  CD,  several  officers  expressed 
concern  about  the  need  for  a  relatively  sophisticated  computer  system  to  run  this  software.  The 
computers  they  were  using  did  not  support  it  adequately,  and  they  thought  other  officers  might 
encounter  similar  problems  in  the  field. 

All  of  the  facilitators  in  the  IMPACT  condition  indicated  they  were  likely  to  incorporate 
elements  of  the  DMC  in  their  future  AARs  (M  =  3.67,  SD  =  .82,  on  a  5-point  scale  where  “5” 
was  labeled  “very  likely”  and  “1”  was  labeled  “not  at  all  likely”).  When  questioned  about  the 
key  differences  between  the  DMC  and  traditional  AARs,  the  majority  of  the  officers  made 
comments  about  the  DMC’s  increased  focus  on  decision  factors  rather  than  tactics  or  procedures. 

As  a  manipulation  check  we  included  a  question  on  the  IMPACT  Facilitators  Post-Test 
Questionnaire  that  asked  them  whether  or  not  they  used  the  DMC  when  facilitating  their  DMG 
sessions.  Five  of  the  six  officers  indicated  they  used  most  of  the  DMC  questions  and  one 
reported  he  used  a  similar  approach  but  did  not  use  any  of  the  specific  questions  provided  in  the 
DMC.  Unfortunately,  this  self-reported  use  of  the  DMC  did  not  correspond  to  the  observations 
made  by  the  research  team  members  who  observed  some  of  the  DMG  sessions.  The  officer  who 
reported  he  did  not  use  the  DMC  questions  did  incorporate  the  gist  of  these  questions  in  his 
debrief,  whereas  another  officer  who  said  he  used  the  DMC  questions  failed  to  incorporate  any 
of  the  questions  in  his  debrief.  This  lack  of  correspondence  between  self-reported  use  of  the 
DMC  and  our  observations  of  the  officers’  actual  performance  as  facilitators  leads  us  to  question 
the  effectiveness  of  our  IMPACT  versus  NO-IMPACT  manipulation. 
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Table  6 


Facilitator  Responses  to  the  Facilitator  Post-Test  Questionnaire  for  IMPACT  and  NO-IMPACT 
Conditions 


Rating  Scale 

Condition 

M 

SD 

1 .  The  DMG  sessions  I  conducted  were  valuable 

IMPACT 

3.83 

1.17 

learning  experiences  for  the  cadets  who 
participated. 

NO-IMPACT 

4.00 

.00 

2.  Participation  in  the  DMG  session  like  the  ones  I 

IMPACT 

3.83 

.75 

conducted  should  improve  the  participants’ 
ability  to  make  rapid  decisions  on  the  battlefield. 

NO-IMPACT 

4.00 

.00 

3 .  DMG  sessions  provide  participants  with  valuable 

IMPACT 

3.33 

1.03 

opportunities  to  practice  their  communications 
skills. 

NO-IMPACT 

3.67 

.52 

4.  DMG  sessions  provide  cadets  with  valuable 

IMPACT 

4.17 

.75 

opportunities  to  practice  their  decision-making 
skills. 

NO-IMPACT 

4.17 

.41 

5.  Participating  in  the  DMG  sessions  made  me 

IMPACT 

4.33 

.52 

think  about  the  many  different  factors  involved 
in  making  rapid  decisions  in  MOUT  settings. 

NO-IMPACT 

4.50 

.55 

6.  The  DMG  sessions  provided  a  great  context  for 

IMPACT 

3.50 

1.22 

me  to  share  my  knowledge  of  battlefield  tactics 
with  a  group  of  cadets. 

NO-IMPACT 

4.08 

.91 

7.  As  the  facilitator  I  was  able  to  keep  all  of  the 

IMPACT 

3.50 

1.22 

cadets  actively  involved  during  the  DMG 

sessions. 

NO-IMPACT 

4.08 

.49 

8.  As  the  facilitator  I  was  able  to  focus  the  group’s 

IMPACT 

3.67 

.82 

discussion  on  the  critical  factors  that  influence 
rapid  decision  making  in  battlefield  settings. 

NO-IMPACT 

4.17 

.41 

9.  I  am  likely  to  use  DMGs  in  the  future  to  improve 
the  decision-making  skill  of  my  subordinates. 

IMPACT 

4.00 

.63 

NO-IMPACT 

4.00 

.63 

10.  DMG  sessions  demonstrated  that  other  cadets 

IMPACT 

4.50 

.55 

can  look  at  the  same  situation  and  come  up  with 
very  different  (but  reasonable)  courses  of  action. 

NO-IMPACT 

4.92 

.20 

1 1. 1  would  have  been  more  confident  facilitating 

IMPACT 

3.33 

DMG  sessions  focused  on  traditional  battlefield 
scenarios  instead  of  MOUT  scenarios. 

NO-IMPACT 

2.92 

Note.  No  significant  differences  were  found. 
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DISCUSSION 


This  preliminary  evaluation  study  was  conducted  in  order  to  obtain  feedback  from  Army 
officers  about  the  usability  of  IMPACT  and  to  better  understand  methodological  issues  involved 
in  measuring  decision  quality.  We  were  particularly  interested  in  whether  Army  officers  would 
be  able  to  leam  to  use  the  DMC  to  facilitate  DMG  sessions  based  on  the  training  they  received 
from  IMPACT  alone  (without  participating  in  a  DST  workshop).  We  also  wanted  to  further 
develop  strategies  to  measure  decision  quality  in  order  to  compare  the  decision-making 
performance  of  individuals  who  participated  in  DMG  sessions  led  by  an  IMPACT-trained 
facilitator  (the  IMPACT  groups)  to  individuals  who  participated  in  DMG  sessions  led  by 
facilitators  who  had  not  had  access  to  IMPACT  (the  NO-IMPACT  groups).  We  reasoned  that  if 
we  had  successfully  implemented  our  DST  program  in  IMPACT,  then  we  should  see  improved 
decision-making  performance  from  individuals  who  participated  in  DMG  sessions  led  by 
facilitators  who  used  IMPACT.  The  results  of  this  study  are  mixed:  it  appears  that  IMPACT  is  a 
very  usable  training  tool,  but  this  particular  study  was  unable  to  demonstrate  that  it  improves  the 
decision-making  performance  of  trainees.  Regarding  methodological  issues,  we  were  able  to 
develop  an  objective  test  to  measure  decision  quality,  as  well  as  a  process  that  resulted  in  reliable 
ratings  of  decision  quality  from  subject-matter  experts. 

Usability  of  IMPACT 

The  officers  who  participated  in  the  DMC  condition  were  asked  to  evaluate  the 
usefulness  of  the  IMPACT  CD  and,  in  general,  had  very  favorable  responses.  The  average 
“usefulness”  rating  of  the  different  instructional  modules  was  a  “4”  on  a  5-point  scale  where  5 
was  labeled  “extremely  useful”  and  1  was  labeled  “not  at  all  useful.”  All  six  of  the  facilitators  in 
the  DMC  condition  indicated  they  were  likely  to  incorporate  elements  of  the  DMC  in  their  future 
AARs.  In  general,  the  feedback  from  the  officers  in  the  DMC  condition  suggests  the  IMPACT 
CD  is  a  easy-to-use  training  tool  which  teaches  Army  officers  how  to  facilitate  DMGs  using  a 
debriefing  technique  that  focuses  trainees’  attention  on  the  cognitive  aspects  of  decision  making 
(e.g.,  the  DMC). 

Although  trainers  reported  that  IMPACT  is  a  valuable  and  highly  usable  training  tool,  we 
found  few  statistically  significant  differences  between  trainees  in  different  conditions.  The 
discussion  below  explores  possible  reasons  for  the  lack  of  measurable  differences  in  decision 
quality. 


There  are  many  reasons  why  we  did  not  observe  performance  differences  between  the 
cadets  in  the  IMPACT  and  NO-IMPACT  conditions.  Of  course  it  is  possible  that  the  lack  of 
observed  differences  reflects  the  fact  that  the  DST  approach  implemented  in  IMPACT  is  not  an 
effective  approach  for  training  recognitional  decision-making  skills.  However,  we  caution  the 
reader  against  interpreting  null  findings  in  this  manner.  There  are  two  other  explanations  for  the 
observed  lack  of  differences  between  groups.  First,  the  manipulation  of  the  IMPACT  versus  NO¬ 
IMPACT  training  was  not  very  strong.  Second,  the  participants  in  the  evaluation  study  may  not 
have  been  the  most  appropriate  sample  for  testing  the  effectiveness  of  IMP ACT/DST. 
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DST  is  not  meant  to  be  a  “quick  fix”  type  of  training  approach  (Pliske  et  al,  2001). 
Ideally,  DST  is  implemented  jfrom  the  top  down  in  organizations.  Supervisors  would  learn  how 
to  use  tools  such  as  the  DMC  to  share  their  domain-relevant  expertise  with  their  subordinates.  It 
takes  time  for  supervisors  to  learn  how  to  use  tools  like  the  DMC.  In  most  of  our  previous  DST 
efforts,  we  have  held  multi-day  workshops  in  which  trainers  get  opportunities  to  practice  using 
the  DST  tools  and  are  provided  feedback  from  experienced  facilitators.  Ideally,  after  trainers 
become  proficient  at  using  the  DST  tools  they  would  then  use  them  with  their  subordinates 
within  the  context  of  their  jobs. 

The  officers  who  participated  as  the  trainers  in  our  evaluation  study  at  USMA  had 
only  a  few  hours  available  to  review  IMPACT  prior  to  conducting  their  DMG  sessions.  Several 
of  the  officers  reported  that  their  computer  systems  did  not  fully  support  the  IMPACT  CD,  which 
could  have  diminished  its  effectiveness  as  a  training  tool.  They  had  no  opportunity  to  practice  the 
new  skills  they  learned  about  on  the  CD,  nor  did  they  have  an  opportunity  to  get  feedback  on 
how  to  effectively  use  tools  such  as  the  DMC.  Most  of  the  officers  had  only  limited  experience 
in  the  MOOT  domain  and  may  not  have  had  a  great  deal  of  expertise  to  share  with  their  trainees. 
Although  all  the  officers  reported  experience-facilitating  groups,  they  were  not  trainers. 
Therefore,  they  may  not  have  been  particularly  motivated  to  improve  their  training  skills.  The 
cadets  who  participated  as  the  trainees  in  our  evaluation  study  had  very  little  prior  military 
experience  and  had  even  less  sophistication  with  MOUT.  And  perhaps,  most  importantly,  the 
officers  and  the  cadets  were  not  part  of  operational  units,  which  may  also  have  affected  their 
motivation  to  learn  how  each  other  “thirdcs”  in  battlefield  situations. 

These  limitations  of  our  sample  may  have  contributed  to  the  lack  of  differences  we 
observed  in  the  performance  of  the  IMPACT  and  NO-IMPACT  groups.  However,  the  most 
serious  limitations  to  the  implementation  of  DST  in  the  evaluation  study  is  that  the  cadets  in 
the  IMPACT  groups  had  only  two  opportunities  to  participate  in  DMG  training  sessions  with 
facilitators  who  had  been  trained  to  use  the  DST  approach.  The  facilitators  were  most  likely  still 
getting  used  to  using  the  DMC  and  leading  the  role-play  that  is  critical  to  the  DST  approach. 
Furthermore,  while  the  three  DMGs  represented  the  same  general  level  of  difficulty,  each  DMG 
addressed  distinct  MOUT  decisions  and  challenges.  Therefore  it  was  not  possible  for  cadets  to 
apply  lessons  they  learned  from  one  DMG  to  their  decisions  on  the  following  DMG(s).  More 
importantly,  it  may  not  be  possible  to  improve  recognitional  decision-making  skills  within  only 
two  training  sessions.  Unfortunately,  the  cadets  at  USMA  were  only  available  for  participation  in 
a  maximum  of  three  55-minute  sessions,  which  is  why  they  only  had  the  opportunity  to  receive 
two  training  sessions  prior  to  the  test  session. 

Although  we  were  unable  to  demonstrate  reliable  differences  between  the  cadets  in  the 
IMPACT  and  NO-IMPACT  conditions  in  terms  of  the  performance  measures  (the  MOUT 
Knowledge  Test  and  the  test  day  DMG  Response  Forms),  we  were  able  to  demonstrate  a  reliable 
difference  in  the  DMG  Session  Evaluation  Forms.  Given  the  limitations  of  the  study  discussed 
above,  this  is  somewhat  surprising.  It  appears  that  in  general,  cadets  in  the  IMPACT  conditions 
had  a  more  positive  learning  experience  than  cadets  in  the  NO-IMPACT  conditions. 
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Methodological  Issues 

Researchers  who  study  naturalistic  decision  making  (NDM)  have  struggled  with  how  to 
operationalize  measures  of  decision-making  quality.  Because  the  NDM  perspective  does  not 
accept  normative  models  of  decision  making  (such  as  Bayes’  Theorem)  as  being  appropriate  for 
evaluating  decision  making  in  the  real  world,  NDM  researchers  are  left  without  a  mathematical 
criterion  to  judge  the  “goodness”  of  a  decision  maker’s  performance.  The  most  typical  solution 
to  this  problem  is  to  have  SMEs  rate  the  quality  of  the  decision  maker’s  performance.  We  have 
stmggled  in  the  past  to  develop  a  process  that  would  result  in  reliable  ratings  fi'om  SMEs  who 
often  have  somewhat  different  ideas  about  what  constitutes  “good”  decision  making  for  a 
specific  domain  relevant  problem.  The  process  used  in  the  present  study  resulted  in  extremely 
high  inter-rater  reliabilities  and  will  be  useful  in  future  research  efforts. 

Another  innovation  included  in  the  present  evaluation  study  was  the  development  and 
use  of  a  multiple-choice  test  of  the  participants’  recognitional  decision-making  skill  within  a 
particular  domain.  We  developed  the  MOUT  Knowledge  Test  (Appendix  D)  used  in  the  present 
study  based  on  the  results  of  our  Cognitive  Task  Analysis  in  this  domain.  With  the  help  of  SMEs 
we  were  able  to  develop  a  scoring  key  for  this  objective  test.  This  approach  to  measuring 
recognitional  decision-making  skill  appears  to  be  very  promising  and  should  be  examined  in 
future  research  efforts. 

The  feedback  we  received  from  the  officers  about  the  usefulness  and  usability  of 
IMPACT  was  very  positive.  They  foimd  the  CD  “clean  and  easy  to  understand,”  and  most 
commented  that  the  video  clips  were  most  helpful  for  providing  examples  of  good  and  bad 
facilitation  techniques.  Many  noted  the  enhancement  that  the  DMC  offers  to  current  AAR 
practices,  and  indicated  that  they  would  use  these  methods  in  the  future.  Some  indicated  that 
the  study  was  a  “good  learning  event”  for  them  and  that  they  enjoyed  participating  in  the 
research.  One  officer  noted  that  “DMGs  are  a  great  tool  and  should  be  made  available  to  all 
cadets  and  leaders  in  the  Army!” 

In  sum,  our  results  suggest  that  we  have  succeeded  in  developing  a  CD-ROM  version 
of  our  DST  program  that  Army  officers  find  usable  and  that  shows  them  how  to  facilitate  DMGs 
using  the  DMC.  In  addition,  important  developments  in  measuring  decision  quality  were  made. 
However,  it  is  not  clear  that  instructors  using  the  techniques  in  IMPACT  can  produce  better 
learning  on  the  part  of  students.  It  may  be  that  IMPACT  is  not  as  powerful  as  the  train-the-trainer 
workshop.  It  is  also  possible  that  the  non-significant  findings  are  a  result  of  artificialities  in  the 
evaluation  environment.  The  cadets  in  the  IMPACT  condition  only  had  the  opportunity  to 
participate  in  two  DMG  sessions,  which  may  not  have  been  enough  time  to  result  in  improved 
recognitional  decision  skills.  Another  artificiality  introduced  by  the  design  of  the  evaluation 
study  was  that  the  DMG  sessions  were  implemented  differently  than  they  would  normally  be, 
with  more  emphasis  on  recording  courses  of  action  than  critiquing  time-pressured  decision 
making.  In  addition,  cadets  may  not  have  perceived  the  DMG  sessions  as  particularly  relevant 
since  they  are  not  currently  facing  MOUT  decisions.  Furthermore,  the  random  tissignment  of 
participants  to  sessions  may  preclude  the  urgency  to  improve  skills  across  team  members  who 
rely  on  one  another,  as  we  have  witnessed  when  DST  is  administered  within  an  operational  unit. 
Further  evaluation  efforts  that  resolve  the  limitations  of  the  current  study  are  necessary  to 
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determine  whether  the  MOUT  DST  program  is  effective  in  improving  decision-making 
performance. 
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APPENDIX  A 


DECISION  MAKING  GAMES  FOR  DAYS  1, 2,  AND  3 


A-l 


I. 


Situation 


Task  Force  Hardnose,  1®*  Battalion,  5*’  Infantry  Division  (Light)  has  been  deployed  as  part  of  a  U.N. 
humanitarian  effort  to  provide  relief  to  the  war-ravaged  country  of  Nubania.  Nubania  has  several  towns  of 
impoverished,  hungry  civilians,  many  of  who  have  been  left  homeless  by  the  war  that  has  recently  ended. 
The  task  force’s  broad  mission  is  as  it  has  been  for  the  three  months  that  they  have  been  deployed  in 
Nubania;  To  maintain  order  and  discipline  and  to  provide  a  security  presence. 

All  villages  in  the  area  are  in  the  same  general  condition.  The  towns  are  in  a  state  of  impoverished  decay. 
Homes  are  simple.  They  are  all  basically  constructed  of  plywood  and  are  one  story.  The  structure  is  the 
same  for  most  dwellings:  four  small  rooms,  each  room  often  packed  full  of  residents.  Most  street  blocks 
have  four  homes  -  two  per  side  backing  two  others.  Separating  the  backyards  of  the  houses  are  rickety, 
6-foot,  wooden  fences.  The  local  marketplace  consists  of  several  small  stands  covered  by  tent-tops, 
where  elderly  women  sell  food  and  clothing. 

A.  Enemy  Forces  (Intelligence  Reports) 

Corbia’s  local  police  force  is  virtually  non-existent  since  order  is  only  now  being  restored  to  this  region. 
You  know  many  of  the  civilians  now  fairly  well,  and  they  are  familiar  with  you.  Most  seem  to  appreciate 
your  presence,  although  a  few  malcontents  are  not  uncommon.  You  have  a  pretty  decent  understanding 
of  who  the  more  troublesome  civilians  are  in  this  town,  since  you  have  been  there  for  so  long.  Since  the 
police  force  is  not  a  presence  to  speak  of,  you  have  been  assigned  the  additional  duty  of  policing  the 
town,  quelling  disturbances,  and  dealing  with  domestic  disputes.  The  civilians  have  access  to  few 
weapons.  Occasionally,  you  must  confiscate  a  Molotov  cocktail,  a  grenade,  or  an  old  semi-automatic  rifle 
from  a  rabble-rouser,  but  so  far,  no  one  in  your  platoon  has  been  injured. 

The  weather  is  hot  and  humid,  winds  are  variable,  and  the  temperature  is  at  least  90  F. 

B  Friendly  Forces 


1®‘  Battalion,  5"’  Infantry  performs  security  operations  in  Sector  Zulu  to  prevent  civilian  unrest  and  ! 

maintain  peace  until  Nubania  reorganizes  its  police  force. 

II.  Mission 

Bravo  company  secures  Corbia,  Tranganu,  and  Finki  in  order  to  prevent  civilian  unrest  and  maintain 
peace  until  Nubania  reorganizes  its  police  force. 

ill.  Execution 

A.  Commander’s  Intent 

I  want  each  village  tightly  controlled  to  prevent  civilian  casualties.  Ensure  all  weapons  are  promptly 
confiscated  and  destroyed  per  SOP.  Keep  good  intelligence  on  troublemakers,  but  attempt  to  prevent 
provocation.  Evacuate  any  casualties  promptly  to  the  Battalion  Aid  Station. 

B.  Concept  of  the  Operation 

1®*  platoon  secures  Tranganu  in  order  to  prevent  civilian  unrest  and  maintain  peace  until  Nubania 
reorganizes  its  police  force.  2"'*  platoon  secures  Corbia  in  order  to  prevent  civilian  unrest  and 
maintain  peace  until  Nubania  reorganizes  its  police  force.  3"^  platoon  secures  Finki  in  order  to 
prevent  civilian  unrest  and  maintain  peace  until  Nubania  reorganizes  its  police  force. 


IV.  Rules  of  Engagement 

Nothing  in  these  ROE  limits  your  right  to  take  appropriate  action  to  defend  yourself  and  your  unit. 

a.  You  have  the  right  to  use  force  to  defend  yourself  against  attacks  or  threats  of  attack. 

b.  Hostile  fire  may  be  returned  effectively  and  promptly  only  to  promote  the  safety  of 
peacekeeping  forces. 

c.  U.S.  forces  should  use  the  minimum  force  necessary  under  the  circumstances  and 
proportional  to  the  threat. 

d.  You  may  seize  property  of  others  to  accomplish  your  mission. 

e.  Detention  of  civilians  is  authorized  for  security  reasons  or  self-defense. 

_ f.  Placing  fire  on  religious  or  cultural  centers  or  towards  schools  or  hospital  is  forbidden. 

V.  Scenario 

Your  platoon  is  conducting  an  on-foot  patrol  of  Corbia.  You  are  walking  west  on  Oak  Street  talking  with 
the  occasional  civilian  who  approaches  you.  Some  local  children  follow  you  and  your  men,  begging  for 
MREs  and  candy.  “Another  boring  day  in  paradise,”  you  think  to  yourself,  as  you  continue  your  patrol. 

Suddenly,  you  hear  gunfire  coming  from  about  100  yards  north  of  your  current  position.  A  civilian  runs  up 
to  you  and  tells  you  that  Alki  has  shot  and  killed  his  wife  in  an  argument  and  has  also  shot  into  the 
intersection,  hitting  two  of  his  neighbors.  You  know  Alki  to  be  one  of  the  town  drunks.  “Where  the  hell  did 
he  get  a  semi-automatic?”  you  think.  At  the  same  time,  you  remember  that  this  citizen  also  has  two  small 
children  who  are  likely  in  the  house.  You  radio  the  company  commander  who  tells  you  it  is  imperative  to 
immediately  get  the  situation  under  control  because  the  battalion  commander  said  and  then  the  radio 
goes  out.  Further  efforts  to  reach  the  company  commander  are  fruitless.  You  are  currently  5  km  from  the 
company  assembly  area  and  consequently  5  km  from  other  known  friendly  units. 

A  crowd  from  the  marketplace  is  now  looking  on  in  fear.  They  all  expect  you  to  restore  order  to  their 
fragile  existence.  You  hear  Alki  cry  out  in  his  native  tongue.  A  local  citizen  translates:  “I’m  fed  up.  I’m 
going  to  kill  everyone!”  As  you  eye  the  situation  from  a  distance,  gunfire  from  the  house  suddenly  sprays 
down  B  street  and  hits  at  least  two  people  from  the  marketplace,  not  to  mention  your  platoon  sergeant, 
grazing  him  in  the  shoulder. 

What  do  you  do? 


VI.  Requirement 

In  a  time  limit  of  2  minutes,  determine  what  course  of  action  you  will  take  and  issue  your  plan  in  the  form 
of  orders  to  your  squad  leaders. 
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I.  Situation 


The  small,  allied  European  nation  of  Klinger  has  been  invaded  by  the  neighboring  country,  Sandia.  The 
Sandians  have  ruthlessly  pushed  south,  through  countless  small  villages,  in  an  effort  to  reach  the  capital 
city  of  Kaytown.  Task  Force  Thunder,  1®*  Battalion,  5“*  Infantry  Division  (Light)  has  been  deployed  as  part 
of  a  U.N.  fighting  force  to  repel  the  invaders.  The  U.N.  force  has  succeeded  in  halting  the  advance  of  the 
invading  enemy,  and  is  currently  pushing  them  back  toward  their  national  boundary. 

A.  Enemy  Forces  (Intelligence  Reports) 

Sandian  forces  are  comprised  of  mostly  light  infantry  and  lightly  armored  wheeled  vehicles,  operating  in 
platoon  and  squad  size  elements  in  your  sector.  They  are  armed  with  7.62  mm  machine  guns  and  have 
light  mortar  capability,  but  the  most  common  weapon  is  the  AK-47.  They  do  have  limited  grenades  and 
light  anti-armor  weapons,  with  some  RPGs.  They  will  usually  avoid  an  all  out  engagement,  but  have 
been  known  to  engage  platoons  in  sector  and  try  to  attrite  forces  through  sniper  attacks  and  ambushes. 

The  weather  is  hot  and  humid  and  the  temperature  is  85F. 

B.  Friendly  Forces 

1®*  Battalion,  S"’  Infantry  attacks  in  zone  to  destroy  enemy  forces  in  order  to  allow  Klinger  forces  to 
defend  Kaytown. 

II.  Mission 

Bravo  Company  attacks  to  destroy  enemy  forces  in  Zone  Lima  in  order  to  allow  Alpha  Company  to  seize 
the  Kaytown  Expressway  Bridge  to  prevent  Sandian  forces  from  attacking  Kaytown. 

III.  Execution 

A.  Commander’s  Intent 

I  want  all  villages  in  our  zone  cleared  of  Sandian  forces.  Search  each  building  and  village  thoroughly. 
Don’t  leave  any  building  unchecked.  I  don’t  want  any  enemy  forces  to  be  able  to  impact  Alpha 
Company’s  attack  on  the  bridge  or  to  linger  behind  if  we  have  to  pull  out  of  sector. 

B.  Concept  of  the  Operation 

platoon,  main  effort,  clears  the  village  of  Garbanza  in  order  to  prevent  enemy  forces  from  attacking 
Alpha  Company  from  the  east.  2"'*  platoon  clears  Hampton  in  order  to  prevent  enemy  elements  from 
setting  up  ambushes  along  the  Garbanza/Hampton  road.  platoon  clears  the  village  of  Mordozi  to 
prevent  enemy  forces  from  infiltrating  along  the  riverbank.  Priority  of  mortar  support  to  1®*  platoon. 


IV.  Rules  of  Engagement 

Nothing  in  these  ROE  limits  your  right  to  take  appropriate  action  to  defend  yourself  and  your  unit. 

a.  In  this  high-intensity  conflict,  you  have  the  right  to  use  force  to  take  appropriate  action  to  defend 
yourself  and  your  unit,  and  to  achieve  your  mission. 

b.  Hostile  fire  may  be  returned  effectively  and  promptly  to  stop  a  hostile  act. 

c.  U.S.  forces  can  use  any  force  deemed  necessary  under  the  circumstances  and  proportional  to  the 
threat. 

d.  Detention  of  civilians  is  authorized  for  security  reasons,  self-defense,  or  to  support  achievement  of 
_ the  mission. 

V.  Scenario 

Your  company  has  been  moving  north,  conducting  clearing  operations  through  several,  war-ravaged, 
deserted  villages  as  you  make  your  way  forward.  Your  company  commander  has  entered  the  village  of 
Hampton  ahead  of  you  with  1®*  and  3*^^  platoons,  expecting  to  pass  through  unimpeded  and  clear  the 
northern  villages  of  Garbanza  and  Mordozi  that  will  allow  Alpha  Company  to  seize  the  critical  bridge. 
However,  you  hear  on  the  radio  that  the  company  is  encountering  unexpected  enemy  resistance.  The 
company  commander  estimates  the  enemy  strength  at  company-level.  He  calls  back  ordering  you  to 
reinforce  the  rest  of  the  company  at  the  corner  of  Elm  and  Banner. 

You  are  just  entering  Hampton  from  the  south  as  you  get  the  call  from  the  Captain.  Your  platoon  is 
traveling  in  HUMVEEs.  As  you  enter  the  village,  you  see  that  Hampton  is  comprised  of  about  15  brick 
and  wooden  buildings  ranging  from  one  to  three-stories.  The  buildings  are  in  varied  states  of  decay.  You 
are  with  1®*  squad  in  the  lead,  heading  north  on  Banner.  2"**  and  3*^  squads  follow  behind.  Your  platoon 
sergeant  is  with  3^*^  squad. 

You  proceed  through  the  intersection  of  Banner  and  Carter  unimpeded,  and  continue  toward  your 
company.  As  you  look  back,  you  observe  that  2"'*  squad  is  now  through  the  intersection.  Suddenly,  3"^^ 
squad  comes  under  attack  from  the  east.  Judging  from  the  small  arms  fire,  you  estimate  an  enemy  force 
of  5-10  enemy  soldiers.  Again  from  the  east,  someone  launches  a  rocket  grenade  that  hits  one  of  the 
HUMVEEs.  Almost  immediately  you  get  a  call  from  3"^  squad  leader.  “We’ve  got  two  serious  injuries  here 
and  one  disabled  HUMVEE.  I  could  use  some  help!”  The  Captain  radios  in,  too.  He  tells  you  they  have 
engaged  the  enemy  and  traded  several  casualties.  He  is  in  need  of  immediate  assistance.  Judging  from 
the  map,  they  are  still  about  1  km  away. 

Meanwhile,  3"^  squad  is  now  stopped  in  the  middle  of  the  intersection,  and  you  have  no  radio  comms 
with  them.  They  continue  to  take  a  steady  stream  of  fire,  and  have  exited  their  vehicles  to  use  them  for 
cover  as  they  return  fire.  A  member  of  1®^  squad  yells  to  you  that  he  thinks  he  saw  movement  in  an  upper 
story  window  in  the  building  to  your  left. 

What  do  you  do? 


VI.  Requirement 


ie 


Difficulty  Level 


iti 


A  Deadly  Approach 


Intermediate 


Scenario  Text  | _ 

Task  Organization:  Bravo  Company,  1^'  Battalion,  Infantry  (Light) 
1®‘  Platoon 
2"'*  Platoon 
3^*^  Platoon 

I  You  are  the  leader  of  2"'*  Platoon,  Bravo  Company. 
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I.  Situation 

In  the  past  six  months,  the  province  of  Amwellia  has  suffered  terrorist  activities  of  increasing  magnitude, 
most  of  which  have  been  aimed  at  the  Muslim  population  and  their  religious  sites.  The  Kona,  a  semi- 
organized  rebel  faction  from  a  neighboring  province,  has  all  but  admitted  to  the  terrorism.  In  response  to 
the  attacks,  the  Amwellian  Muslims  have  organized  a  military  group  that  has  carried  out  its  own  terrorist 
activities  aimed  at  the  Kona.  The  Muslims  call  themselves  the  Butri. 

Two  weeks  ago,  after  escalation  of  terrorism  by  both  sides,  the  Kona  invaded  the  Amweiiian  city  of 
Botswell.  Botswell  has  a  population  of  about  20,000,  about  half  of  whom  are  Muslims.  Since  the  initial 
invasion,  the  Kona  and  Butri  have  been  vying  for  controi  of  the  city,  neither  group  able  to  establish  and 
maintain  a  strong  foothold. 

Task  Force  Strongarm,  1®*  Battalion,  5®’  Infantry  (Light)  has  been  deployed  as  part  of  a  U.N.  humanitarian 
effort  in  Amwellia.  As  a  result  of  the  fighting  the  city  has  been  cut  off  from  the  rest  of  the  country,  and 
civilians  are  beginning  to  feel  the  food  and  supply  shortages.  The  UN  is  responding  by  initiating  relief 
efforts  to  Botswell. 

There  are  two  major,  armed  factions  fighting  for  controi  of  this  high-desert  country.  Violent  clashes  with 
Soviet-weapons— typically  AK-47s  and  RPGs  and  truck-mounted  machine  guns— are  not  uncommon. 

The  factions  are  very  difficult  to  distinguish,  and  often  they  appear  no  different  from  the  local  civilians. 

A.  Enemy  Forces  (Intelligence  Reports) 

The  Kona  are  a  group  of  a  few  hundred  men,  most  of  whom  are  armed  with  rifles  and  hand  grenades.  A 
few  have  been  seen  with  machine  guns  and  RPGs.  They  tend  to  move  in  groups  of  4-5.  It  is  unclear  who 
their  leader  is,  and  how  well  they  are  organized.  Their  communications  gear  seems  to  be  primitive  and 
basic,  and  on  a  good  day  the  groups  of  5-6  men  will  have  one  radio  amongst  them.  The  Kona  tend  to 
fight  only  if  they  have  the  upper  hand,  but  when  they  do,  they  attack  fiercely  to  seize  control  of  a  building 
or  area. 

The  Butri  seem  to  have  more  men  than  the  Kona,  but  are  less  well  equipped.  Their  weapon  types  are  the 
same  as  the  Kona,  but  they  have  fewer  of  them,  and  sometimes  not  enough  to  arm  all  their  soldiers.  Their 
operations  are  only  semi-organized,  and  mostly  defensive  in  nature.  They  tend  to  dig  into  key  areas  of  the 
city  to  prevent  the  Kona  from  taking  those  areas. 

The  civilians  in  Botswell  are  clearly  siding  with  the  Butri,  but  not  to  the  extent  that  they  will  join  in  the 
fighting.  They  tend  to  stay  close  to  their  homes  as  much  as  possible,  however  it  is  not  unlikely  to  see 
groups  of  2  or  3  walking  briskly  down  the  streets  to  the  market.  Some  of  the  more  affluent  civilians  have 
evacuated  the  town,  but  about  70%  remain. 

The  city  of  Botswell  covers  approximately  10  square  miles.  It  is  situated  in  a  slight  valley,  with  rolling  hills 
on  the  outskirts  of  the  city.  Vegetation  is  desert-like,  with  brush  and  occasionaliy  small  trees  dotting  the 
landscape.  Within  the  city,  trees  have  been  preserved  but  brush  has  been  removed.  The  roads  within  the 
city  are  mostly  dirt,  with  a  few  major  roads  paved  with  cement  slabs.  Almost  all  roads  are  12  feet  wide, 
the  major  ones  perhaps  14  feet  wide.  The  buildings  in  Botswell  vary  quite  a  bit  in  terms  of  their 
construction.  Most  are  one-story,  with  a  few  two-  and  three-story  buildings  located  in  the  central, 
“downtown”  area.  The  residential  buildings  are  generally  made  of  plaster  or  wood,  while  the  more 
commercial  and  governmental  buildings  tend  to  be  made  of  brick  or  concrete  blocks.  Across  the  street 
from  the  town  hall  and  in  the  area  surrounding  the  mosque  are  sacred  courtyards  contained  by  2-feet 
high  stone  walls. 
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Intel  has  reported  that  there  has  been  a  significant  amount  of  unrest  in  Amwellia  in  the  past  week.  Small 
bands  of  5-6  men  from  the  factions  will  pass  though  town,  and  have  often  started  skirmishes  or  drawn 
sniper  fire,  resulting  in  significant  civilian  injuries  and  even  deaths.  You  have  been  told  that  you  are 
currently  the  only  peacekeeping  forces  in  the  village. 

The  weather  is  muggy  and  dry,  winds  are  variable,  and  the  temperature  is  80F. 

B.  Friendly  Forces 

1®*  Battalion,  5**’  infantry  provides  security  in  Amwellia  in  order  to  support  U.N.  efforts  to  provide 
humanitarian  relief  to  affected  civilian  populations. 

il.  Mission 

Bravo  Company  provides  security  for  medical  and  food  relief  missions  in  Botswell  to  protect  U.N. 
personnel  and  allow  civilians  free  and  unhindered  access  to  U.N.  medical  and  food  relief  teams. 

III.  Execution 

A.  Commander’s  Intent 

I  want  each  platoon  positioned  vicinity  of  the  relief  operation  sites  in  Botswell  to  provide  security  against 
any  hostile  force  intent  on  disrupting  the  food  and  medical  distribution.  Ensure  that  civilians  have  access 
to  each  site.  Secure  the  site  and  prevent  disruption  from  factional  elements.  Protect  the  U.N.  operation  at 
your  site.  Remain  on  station  to  allow  the  medical  team  to  depart  unharmed  out  of  the  area. 

B.  Concept  of  the  Operation 

1®‘  platoon  secures  Site  East  in  order  to  protect  and  allow  civilian  access  to  U.N.  relief  teams.  2"^  platoon 
secures  Site  Central  in  order  to  protect  and  allow  civilian  access  to  U.N.  relief  teams.  3^^^  platoon  secures 
Site  West  in  order  to  protect  and  allow  civilian  access  to  U.N.  relief  teams.  No  fire  support  is  available. 
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IV.  Rules  of  Engagement 


Nothing  in  these  rules  negates  your  inherent  right  to  use  reasonable  force  to  defend  yourself  against 
dangerous  personal  attack. 

a.  Deadly  force  may  be  used  only  when  fired  upon,  when  clear  evidence  of  hostile  intent  exist,  or 
when  armed  elements,  mobs,  and/or  rioters  threaten  human  life,  sensitive  equipment  and  aircraft, 
or  open  and  free  passage  of  relief  supplies. 

b.  In  situations  where  deadly  force  is  not  appropriate,  use  the  minimum  force  necessary  to 
accomplish  the  mission. 

c.  Unattended  means  of  force  (e.g.,  mines,  booby  traps,  trip  guns)  are  not  authorized. 

d.  U.S.  forces  will  not  endanger  or  exploit  the  property  of  the  local  population  without  their  explicit 
approval. 

e.  Treat  all  persons  with  dignity  and  respect 

f.  Placing  fire  on  religious  sites  is  not  authorized. 


V.  Scenario 

Your  platoon’s  current  mission  is  to  take  control  of  a  key  intersection  in  central  Botswell  and  secure  the 
area  so  that  food  and  medical  supplies  can  be  distributed  to  civilians  from  the  town  hall.  In  order  to 
secure  this  intersection,  you  must  clear  and  take  control  of  the  town  hall,  which  is  on  the  southeast 
corner  of  Manna  and  Holti  Streets.  S-2  reports  do  not  indicate  that  either  faction  occupies  the  town  hall  or 
any  nearby  buildings.  However,  you  have  been  warned  of  their  potential  presence  in  the  area,  given  the 
centrality  of  the  intersection. 

It  is  0100.  The  skies  are  cloudless  and  the  moon  is  a  crescent.  Your  platoon  consists  of  5  fire  teams  with 
3  SAWs.  All  of  your  soldiers  are  equipped  with  NVGs.  Your  plan  is  to  walk  south  down  Manna  Street  and 
set  up  an  overwatch  in  the  two-story  warehouse  on  the  northwest  corner;  the  warehouse  has  second- 
story  windows  that  provide  a  good  view  of  the  town  hall  and  the  entire  intersection.  You  need  to  have  the 
town  hall  cleared  in  time  for  food  and  supplies  to  be  distributed  starting  at  0800  tomorrow. 

As  you  approach  the  warehouse,  you  notice  that  the  back  door  has  seen  better  days  -  there  are  at  least 
three  sizable  dents  in  the  door,  and  the  lock  and  handle  apparatus  look  to  have  been  jimmied.  But  you 
can’t  tell  from  your  current  position  whether  the  perpetrators  were  successful  at  getting  into  the 
warehouse.  You  scan  the  area,  and  spot  two  men  with  rifles  slung,  carrying  large  duffle  bags,  walking 
quickly  but  cautiously  across  Manna  Street,  heading  west  from  the  tovwi  hall.  Then  you  get  a  radio  call 
from  your  platoon  sergeant:  “There’s  some  commotion  over  here  in  the  residential  area.  I  can’t  tell  who 
they  are,  but  I  see  at  least  20  people  moving  around.  Looks  like  they’re  up  to  no  good. 

What  do  you  want  to  do?” 


Requirement 

In  a  time  limit  of  2  minutes,  issue  your  orders  and  intent  to  your  three-squad  leaders.  Include  any 
additional  communications  you  will  make. 
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APPENDIX  B 


HOW  TO  CONDUCT  A  DECISION  MAKING  GAME 


Instructions: 

How  to  Conduct  a  Decision  Making  Game  (DMG) 

IMPACT 


Guidelines  for  Facilitators: 

Over  the  next  two  days  you  will  be  conducting  Decision  Making  Game  (DMG)  seminars. 
This  guide  will  give  you  an  idea  of  what  to  do  in  the  seminars  and  the  order  in  which  to 
do  it. 

There  are  8  general  steps  for  the  seminar.  Below  you  will  see  the  8  steps,  a  brief 
description  of  what  to  do  in  each,  and  a  sample  dialogue  (when  relevant).  The  samples 
are  intended  to  be  used  as  examples  of  possible  wording  for  various  phases  of  the 
seminar.  Please  use  the  samples  as  a  starting  point  in  leading  the  sessions.  We 
encourage  you  to  use  your  own  words  in  facilitating  the  sessions  rather  than  reading  the 
sample  dialogue  to  the  class. 


1.  Introduction 

Give  a  brief 
introduction  to 
the  students 
about  what  they 
wiii  be  doing  over 
the  course  of  the 
next  hour. 


2.  Materials 

Pass  out  the  DMG 
Response  Form 
to  the  students. 
Please  have  them 
write  their  name 
and  the  date  at 
the  top  of  the 
form. 


Sample  dialogue:  “Good  afternoon.  Today  we  will  be  participating 
in  a  study  about  developing  training  for  soldiers  faced  with 
challenges  in  urban  combat.  We  will  be  practicing  making  decisions 
and  choosing  courses  of  actions  in  MOUT  scenarios. 

We  will  be  doing  an  exercise,  called  a  Decision  Making  Game,  or 
DMG.  I  will  read  you  a  scenario  about  a  battlefield  situation.  You  will 
have  a  few  minutes  to  figure  out  your  best  Course  of  Action  ( COA). 
You  will  have  to  write  you  answers  down  NEATLY  and  LEGIBLY  on 
the  DMG  Response  Form.  We’ll  then  discuss  different  individuals’ 
responses  to  the  scenario.  ” 

Sample  dialogue:  ‘This  is  the  DMG  Response  Form  we  will  be 
using.  Please  make  sure  that  you  include  your  name  and  today’s 
date  at  the  top  of  form.  I  want  you  to  have  a  look  at  this  before  we 
start,  so  you  are  familiar  with  the  things  you  will  be  asked  to  write 
when  the  time  comes.  We  will  complete  one  Decision  Making  Game 
today.  You  will  complete  additional  decision  games  over  the  next  two 
days.” 
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3.  Read  The  Scenario 


Read  the  DMG 
scenario.  Use  an 
overhead  projector 
to  show  the  map 
that  corresponds 
to  the  scenario. 
Point  out 
buildings, 
locations,  or 
landmarks  on  the 
map  as  you  read 
about  them. 


Sample  dialogue:  “Let  me  read  you  the  scenario.  This  game  is 

called _ .  (Fill  in  the  name  of  the  appropriate 

Decision  Making  Game.)  You  are  the  leader  of  the  2"“  platoon, 
Bravo  Company...” 


4.  Allow  Participants  To  Ask  Questions 


Ask  the  cadets  if  they  have  any  questions  about  the  scenario.  Only  answer  questions  that  will 
clarify  the  scenario.  Some  questions  may  require  answers  that  provide  additional  information  to 
what  is  written  in  the  scenario.  If  you  choose  to  answer  these  questions,  please  make  sure  your 
responses  are  consistent  with  the  information  presented  in  the  scenario.  You  may  also  reply  to 
any  question,  “You  would  not  have  that  information.” 

5.  Give  The  Students  Time  To  Complete  The  DMG  Response  Form 

Give  the  students  Sample  dialogue:  “You  have  four  minutes  to  write  your  orders 

four  minutes  to  and  assessment  of  the  situation  on  the  DMG  response  form, 

communicate  their  Please  write  as  neatly  and  legibly  as  possible.  ” 

COA  in  the  form  of 
orders.  Remind 
them  to  write 
neatly  and  legibly. 

6.  Allow  Students  To  Present  Their  Responses 

Call  on  a  student  to  state  his/her  orders  as  if  s/he  were  the  platoon  leader  in  this  scenario. 
Assign  the  role  of  squad  leaders  to  other  students  in  the  class. 

Use  the  Decision  Making  Critique  (DMC)  to  facilitate  group  discussion.  Some  of  the  key  steps 
are  reviewed  in  Section  7.  When  asking  questions  from  the  DMC,  put  them  in  words  you  are 
comfortable  with.  If  you  don’t  get  a  response,  try  phrasing  the  questions  in  a  different  way. 
Don’t  feel  like  you  have  to  follow  the  DMC  word-for-word. 
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7.  Conduct  the  Decision  Making  Critique  (PMC) 


Step  1 :  Begin  the 
process  by  having 
participants  identify 
critical  decision  points. 


Step  2:  After  students 
identify  each  tough 
decision,  focus  the 
discussion  to  further 
break  down  each 
decision. 

One  of  the  more 
challenging  aspects 
of  good  facilitating  is 
to  notice  differences 
in  responses  and  to 
make  those 
differences  visible  to 
the  class.  There  is  a 
tendency  for 
participants  to  agree 
too  quickly  that  their 
responses  are 
basically  the  same. 
Don’t  bow  to  this 
pressure.  Take  the 
time  to  pursue  subtle 
differences  in 
responses,  for  they 
almost  always  reflect 
very  different 
perceptions  of  the 
same  situation. 


After  the  orders  are  issued,  start  the  debrief  by  asking  the 
following  question: 

What  was  the  first  tough  decision  you  faced? 

(At  this  point,  it  is  helpful  to  have  the  participant  point  out  decision 
points  on  a  map  display  as  s/he  discusses  them). 

Sample  dialogue:  What  was  the  most  difficult  decision  you 
had  to  make  in  this  situation? 


For  each  critical  decision  ask: 

•  Why  was  it  difficult? 

•  Why  did  you  choose  that  course  of  action? 

•  What  one  piece  of  missing  information  would  have  helped 
you  the  most? 

•  What  other  actions  did  you  consider?  Why  didn't  you 
choose  them? 

Listen  for  disagreement  among  the  class.  Follow  up.  Ask  those 
who  disagree  to  explain  their  positions. 

Sample  dialogue:  “Williams,  you  mentioned  that  the  decision 
to  do  X  was  difficult.  Why  was  that  difficult?”  When  you  ask 
these  questions,  be  patient.  Allow  the  participants  time  to  think, 
and  don’t  get  uncomfortable  with  periods  of  silence.  Continue 
to  probe,  even  after  they  provide  an  initial  answer:  “What  else 
made  that  a  difficult  decision?”  If  they  can’t  think  of  anything 
else,  try  asking,  “How  would  a  brand  new  soldier  mess  up  this 
decision?  Why?” 

Listen  to  parts  of  the  response  that  suggest  difficult  elements, 
and  follow  up  on  those.  Also,  ask  others  in  the  class  to  provide 
their  thoughts  on  these  questions.  The  more  you  can  get  the 
rest  of  the  class  involved,  the  better. 

Ask  another  student  to  state  his/her  orders  as  if  s/he  were 
the  platoon  leader  in  this  scenario  and  repeat  Steps  1  and 
2. 
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step  3:  Use  the 
following  set  of 
questions  to  summarize 
the  lessons  learned 
from  the  training 
exercise. 


Ask: 

•  What  would  you  do  differently  if  you  were  in  this  situation 
again? 

•  What  was  our  biggest  weakness?  What  was  our  biggest 
strength? 

•  What  are  some  important  lessons  learned  from  this 
exercise? 


When  asking  these 
questions,  be  sure  to  ask 
several  trainees  for  their 
perspectives.  You  might 
want  to  write  the  lessons 
on  the  board  to  summarize 
the  exercise. 


Listen  for:  How  class  members  would  make  decisions 
differently  (seek  out  more  information,  check  assumptions,  be 
more  attentive  to  cues). 

Sample  dialogue:  “Based  on  what  we’ve  been  discussing 
here,  if  you  were  placed  in  this  situation  again,  what  wouid  you 
do  differentiy?”  (Give  them  some  time  to  think).  “Why?”  To  the 
class;  “Who  would  do  something  differently?”  Take  some  time 
here  to  probe  this  question. _ 


Tips  for  Facilitating 
the  Decision  Making 
Critique 


>  Try  to  keep  everyone  involved,  but  don’t  get  sidetracked. 
Don’t  stay  on  one  topic  so  long  that  you  lose  class  member 
interest.  Keep  class  members  engaged. 

>  Listen  for  statements  that  will  open  the  door  for  more 
discussion  (e.g.,  “What  I  needed  from  him  was  ...I  just 
assumed  that  they  knew  that  ...I  don’t  know  where  I  would 
find  that  out...”). 

You  want  to  stimulate  class  members  to  ask  questions  of 
themselves  about  what  they  should  have  known  or  seen. 

>  Don’t  get  sidetracked  into  discussions  of  what  was  the  best 
action.  You  are  trying  to  change  the  way  the  class 
members  think  about  things:  the  patterns  they  should  have 
noticed,  the  early  alarm  bells  that  should  have  gone  off. 


Remember  to  reserve  5  minutes  for  the  session  evaluation  form. 


8.  Distribute  Session  Evaluation  Form 


You  will  receive 
envelopes  with 
evaluation  materials 
from  the  Klein 
Associates  repre¬ 
sentatives.  At  the  end 
of  each  DMG  session, 
designate  one  student 
to  pass  out  and 
collect  the  materials 
in  the  envelopes.  To 
ensure  anonymity, 
wait  in  the  hall  while 
the  students  are 
writing.  Then  collect 
the  sealed  envelope 
and  return  it  to  the 
Klein  Associates 
representatives. 


Sample  dialogue:  “Thank  you  all  for  participating.  There  is  one 
more  thing  I  need  you  to  do.  There  are  some  questionnaires  that 
need  to  be  filled  out.  Do  not  put  your  name  on  the  questionnaire, 
but  please  include  the  date  and  my  name,  it  is  very  important 
that  you  be  honest  in  all  of  your  responses.  1  will  not  see  your 
evaluations.  The  people  running  this  study  will  use  them,  for 
research  purposes  only. 

Cadet  Smith,  please  see  that  these  questionnaires  are 
distributed.  When  everyone  is  finished,  collect  both  the 
questionnaires  and  the  DMG  response  forms,  seal  them 
in  the  envelope,  and  hand  them  to  me.  I  will  be  waiting  in  the 
hall.” 
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Instructions: 

How  to  Conduct  a  Decision  Making  Game  (DMG) 

NONIMPACT 


Guidelines  for  Facilitators: 

Over  the  next  two  days  you  will  be  conducting  Decision  Making  Game  (DMG) 
seminars.  This  guide  will  give  you  an  idea  of  what  to  do  in  the  seminars  and  the 
order  in  which  to  do  it. 


There  are  8  general  steps  for  the  seminar.  Below  you  will  see  the  8  steps,  a  brief 
description  of  what  to  do  in  each,  and  a  sample  dialogue  (when  relevant).  The 
samples  are  intended  to  be  used  as  examples  of  possible  wording  for  various 
phases  of  the  seminar.  We  encourage  you  to  use  your  own  words  in  facilitating 
the  sessions  rather  than  reading  the  sample  dialogue  to  the  class. 


1.  Introduction 

Give  a  brief 
introduction  to 
the  students 
about  what  they 
will  be  doing  over 
the  course  of  the 
next  hour. 


Sample  dialogue:  “Good  afternoon.  Today  we  will  be  participating 
in  a  study  about  developing  training  for  soldiers  faced  with 
challenges  in  urban  combat.  We  will  be  practicing  making  decisions 
and  choosing  courses  of  actions  in  MOUT  scenarios. 

We  will  be  doing  an  exercise,  called  a  Decision  Making  Game,  or 
DMG.  I  will  read  you  a  scenario  about  a  battlefield  situation.  You  will 
have  a  few  minutes  to  figure  out  your  best  Course  of  Action  (COA). 
You  will  have  to  write  you  answers  down  NEATLY  and  LEGIBLY  on 
the  DMG  Response  Form.  We’ll  then  discuss  different  individuals’ 
responses  to  the  scenario.  ” 


2.  Materials 

Pass  out  the  DMG 
Response  Form 
to  the  students. 
Please  have  them 
write  their  name 
and  the  date  at 
the  top  of  the 
form. 


Sample  dialogue:  “This  is  the  DMG  Response  Form  we  will  be 
using.  Please  make  sure  that  you  include  your  name  and  today’s 
date  at  the  top  of  form.  I  want  you  to  have  a  look  at  this  before  we 
start,  so  you  are  familiar  with  the  things  you  will  be  asked  to  write 
when  the  time  comes.  We  will  complete  one  Decision  Making  Game 
today.  You  wiil  complete  additional  decision  games  over  the  next  two 
days.” 
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3.  Read  The  Scenario 


Read  the  DMG 
scenario.  Use  an 
overhead  projector 
to  show  the  map 
that  corresponds 
to  the  scenario. 
Point  out  build¬ 
ings,  locations,  or 
landmarks  on  the 
map  as  you  read 
about  them. 


Sample  dialogue:  “Let  me  read  you  the  scenario.  This  game  is 

calied _ (Fiii  in  the  name  of  the  appropriate 

Decision  Making  Game.)  You  are  the  ieader  of  the  2"°  piatoon, 
Bravo  Company. . .  ” 


4.  Allow  Participants  To  Ask  Questions 


Ask  the  cadets  if  they  have  any  questions  about  the  scenario.  Only  answer  questions  that  will 
clarify  the  scenario.  Some  questions  may  require  answers  that  provide  additional  information  to 
what  is  written  in  the  scenario.  If  you  choose  to  answer  these  questions,  please  make  sure 
your  responses  are  consistent  with  the  information  presented  in  the  scenario.  You  may  also 
reply  to  any  question,  “You  would  not  have  that  information.” 

5.  Give  The  Students  Time  To  Complete  The  DMG  Response  Form 

Give  the  students  Sample  dialogue:  “You  have  four  minutes  to  write  your  orders 

four  minutes  to  and  assessment  of  the  situation  on  the  DMG  response  form.  Please 

communicate  their  write  as  neatly  and  legibly  as  possible.  ” 

COA  in  the  form  of 
orders.  Remind 
them  to  write 
neatly  and  legibly. 

6.  Allow  Students  To  Present  Their  Responses 

Call  on  a  student  or  students  to  present  his/her  plan  to  the  group.  Use  the  standard  After  Action 
Review  (AAR)  format  to  facilitate  group  discussion.  Some  of  the  key  steps  in  the  AAR  are 
reviewed  in  Section  7. 
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7.  Conduct  the  After  Action  Review 


Step1:  Establish 
what  happened. 

For  this  DMG  session, 
discuss  the  likely 
outcome(s)  of  the  plan 
generated  and  briefed 
by  an  individual 
student  (or  several 
students). 

The  facilitator  and  the  participants  determine  what 
actually  happened  during  performance  of  the  training 
task. 

Step  2:  Determine 
what  was  right  or 
wrong  with  what 
happened. 

For  this  DMG  session, 
discuss  what  was  right 
or  wrong  with  plan(s) 
briefed  by  students. 

The  participants  establish  the  strong  and  weak  points  of 
their  performance.  The  facilitator  plays  a  critical  role  in 
guiding  the  discussions  so  that  conclusions  reached  by 
participants  are  doctrinally  sound,  consistent  with  Army 
standards,  and  relevant  to  the  wartime  mission. 

Step  3:  Determine  how 
the  task  should  be 
done  differently  the 
next  time. 

For  the  DMG  session, 
discuss  how  the  DMG 
should  be  addressed  If 
it  were  a  real-world 
situation. 

The  facilitator  leads  the  group  in  determining  exactly  how 
participants  will  perform  differently  the  next  time  the  task 
is  performed.  This  results  in  organizational  and  Individual 
motivation  to  conduct  future  sustainment  training  at 
desired  levels  of  proficiency. 

Step  4:  Perform  the 
task  again. 

Your  class  will  have 
an  opportunity  to 
respond  to  another 

DMG  in  Session  2. 

This  is  done  as  soon  as  possible  to  translate  observation 
and  evaluation  into  corrective  action.  Additional  training 
allows  the  participants  to  apply  the  lessons  learned 
during  the  AAR.  Leaders  understand  that  all  tasks  will  not 
be  performed  to  standard.  Therefore,  during  the  short- 
range  and  near-term  planning  process,  they  provide 
flexibility  in  training  events  and  schedules  which  allow  for 
additional  training  immediately  following  the  AAR. 

Remember  to  reserve  5  minutes  for  the  session  evaluation  form. 
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8.  Distribute  Session  Evaluation  Form 


You  will  receive  Sample  dialogue:  “Thank  you  all  for  participating.  There  is  one 

envelopes  with  more  thing  I  need  you  to  do.  There  are  some  questionnaires 

evaluation  materials  that  need  to  be  filled  out.  Do  not  put  your  name  on  the  question- 

from  the  Klein  naire,  but  please  include  the  date  and  my  name.  It  is  very 

Associates  repre-  important  that  you  be  honest  in  all  of  your  responses.  I  will  not 

sentatives.  At  the  end  see  your  evaluations.  The  people  running  this  study  will  use 

of  each  DMG  session,  them,  for  research  purposes  only, 

designate  one  student 

to  pass  out  and  collect  Cadet  Smith,  please  see  that  these  questionnaires  are 
the  materials  in  the  distributed.  When  everyone  is  finished,  collect  both  the 

envelopes.  To  ensure  questionnaires  and  the  DMG  response  forms,  seal  them 

anonymity,  wait  in  the  in  the  envelope,  and  hand  them  to  me.  I  will  be  waiting  in  the 

hall  while  the  students  hall.  ” 

are  writing.  Then 

collect  the  sealed 

envelope  and  return  it 

to  the  Klein 

Associates 

representatives. 


APPENDIX  C 


DMG  RESPONSE  FORMS  AND  TEST  DAY  DMG  RESPONSE  FORMS 
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DMG  Response  Form 
Today’s  Date: _ 


Your  Facilitator: 


YoiarName: 


You  have  approximately  4  minutes  to  write  down  your  responses.  PLEASE  WRITE  LEGIBLY 
so  someone  else  can  read  it. 

1)  State  your  orders  in  the  exact  language  that  you  would  use  to  deliver  them  to  your  squad 
leaders  and  other  attached  units  (if  applicable). 

Squad  leader  1: 


Souad  leader  2; 


Souad  leader  3: 


Additional  units  (if  applicablel; 


2)  State  any  additional  communications  you  would  make  to  people  outside  of  your  platoon. 
Specify  to  whom  you  would  be  communicating.  Note  that  not  all  DMGs  involve  communication 
outside  of  your  platoon. 


3)  Describe  vour  assessment  of  the  situation — ^what  do  you  believe  is  happening  in  this  DMG? 
(You  can  continue  on  the  back  of  this  form  if  you  need  more  room.) 


DMG  Test  Form 

Today’s  Date: _ Your  Facilitator: _ Your  Name:. 


Part  1.  You  have  approximately  3  minutes  to  write  down  your  communications.  State  your 
orders  in  the  exact  language  that  you  would  use  to  deliver  them  to  your  squad  leaders  and  other 
attached  units  (if  applicable).  PLEASE  WRITE  LEGIBLY  so  someone  else  can  read  it. 


Squad  leader  1: 


Squad  leader  2: 


Squad  leader  3: 


Additional  uuits  (if  applicable): 


2)  State  any  additional  communications  you  would  make  to  people  outside  of  your  platoon. 
Speciiy  to  whom  you  would  be  communicating.  Note  that  not  all  DMGs  involve  commrmication 
outside  of  your  platoon. 
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DMG  Test  Form 

Today’s  Date: _ Your  Facilitator: _ Your  Name: _ 

Part  2.  You  have  approximately  5  minutes  to  answer  the  following  questions  about  the  DMG. 
Please  write  legibly  so  someone  else  can  read  it. 

1)  Describe  vour  assessment  of  the  situation — ^what  do  you  believe  is  happening  in 

this  DMG? 


2)  Elaborate  on  vour  plan  that  you  communicate  to  your  squad  leaders  (and  other 
additional  units)  in  Part  1 — ^what  are  you  trying  to  accomplish?  Why  did  you 
choose  this  other  course  of  action? 


3)  What  is  vour  greatest  concern  given  the  situation  described  in  the  DMG? 
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APPENDIX  D 


MOUT  KNOWLEDGE  TEST 


Today’s  date: 


Your  facilitator’s  name: 


MOUT  KNOWLEDGE  TEST 
(Acceptable  Answers  in  Bold) 

This  questionnaire  consists  of  two  scenarios,  each  with  8  questions,  that  require  you  to  make 
judgments  about  the  situation.  You  are  the  platoon  leader.  Read  the  scenario  and  then  answer  the 
questions  that  follow.  Choose  the  best  answer  of  those  available.  Circle  your  choice. 

Scenario  One 

You  are  the  platoon  leader  of  3'^  Platoon,  B  Company  and  are  involved  in  a  high-intensity 
conflict.  Yoiu:  unit  is  part  of  a  brigade  task  force  attempting  to  push  the  invading  Mahrian  army  north, 
toward  their  own  border,  after  their  invasion  of  an  allied  country.  Your  platoon  is  the  rear  element  of  B 
Company.  You  are  in  HMMWVs  mounted  with  M-240  machine  guns  and  have  stopped  momentarily  in  a 
town  that  1“  and  2”^  platoons  cleared  of  enemy  soldiers  4  days  ago.  You  are  currently  in  a  large  one-room 
warehouse,  checking  and  reloading  supplies,  and  communicating  with  key  leaders  in  the  platoon.  and 
2'^  squad  leaders  are  in  the  warehouse  with  you.  3"^^  squad  leader  is  outside  overseeing  the  reloading  of 
supplies  into  the  HMMWVs.  The  machine  gunners  are  outside,  pulling  local  security  on  the  warehouse 
and  your  position.  It  is  nighttime  and  2200,  the  skies  are  overcast,  and  the  weather  is  90°F  and  humid. 

As  part  of  your  current  operation  order,  the  company  commander’s  intent  follows: 

I  want  all  villages  thoroughly  searched  and  cleared  of  all  enemy  elements.  We  cannot  allow 
enemy  forces  to  escape  the  villages  and  conduct  hit  and  run  operations  against  the  battalion ’s 
rear  line  of  communications.  Destroy  all  enemy  forces  in  sector. 

Your  mission  is: 

3”*  Platoon,  B  Company,  clears  Sector  Charlie  in  order  to  keep  battalion  lines  of  communication 
open  along  Route  Zulu. 

Your  ROE  are  the  following: 

Nothing  in  these  ROE  limits  your  right  to  take  appropriate  action  to  d^end  yourself  and  your 
unit. 

a.  In  this  high-intensity  conflict,  you  have  the  right  to  use  force  to  take  appropriate  action  to 
defend  yourself  and  your  unit,  and  to  achieve  your  mission. 

b.  Hostile  fire  may  be  returned  effectively  and  promptly  to  stop  a  hostile  act. 

c.  U.N.  forces  can  use  any  force  deemed  necessary  under  the  circumstances  and  proportional 

to  the  threat. 

Detention  of  civilians  is  authorized for  security  reasons,  self-defense,  or  to  support  achievement 
of  the  mission. 

Your  platoon  has  had  no  enemy  contact  over  5  days.  Your  earlier  engagements  involved  just  a 
few  (4-8)  enemy  soldiers  who  surrendered  when  they  saw  you  approach.  The  company  commander  is 
with  1”  and  2“*  platoons,  2  km  due  north  of  your  position. 

In  the  middle  of  updating  your  squad  leaders,  your  RTO  hands  you  the  radio  iruke.  It’s  the 
company  commander.  He  is  yelling  above  the  sovmds  of  explosions  and  the  soimd  of  machine  gun  fire  in 
the  background.  You  receive  a  transmission  Jfrom  him:  “I  need  you  to  move  immediately  out  of  that  town 
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and  catch  up  with  us!  We’re  taking  heavy  fire  from  . . The  rest  of  his  message  is  garbled  and  then  goes 
dead  before  you  can  respond.  Attempts  to  get  him  back  on  the  radio  are  not  successful. 

You  direct  your  leaders  to  go  to  Redcon  One  for  immediate  movement  north.  However,  the 
reloading  is  only  half  complete.  There  is  still  some  ammunition,  fuel,  and  rations  inside  the  warehouse. 
You  are  inside  the  warehouse  giving  a  Frago  to  your  platoon  sergeant  when  you  hear  gunfire,  AK-47  you 
believe,  erupt  outside,  followed  by  the  unmistakable  sound  of  M- 16s.  3"*  squad  leader  radios  in  to  you: 
“LT,  we  are  taking  enemy  fire  from  the  grocery  store  20  meters  to  our  northeast.  No  one  seems  to  have 
been  hit.  We’ve  taken  cover  behind  the  vehicles  and  in  doorways,  and  have  returned  fire.  From  the  rate  of 
fire.  I’d  say  there  are  about  five  or  six  weapons  trained  on  us.  We’re  not  taking  fire  from  any  other 
buildings,  but  one  of  my  guys  thought  he  just  saw  something  move  in  the  house  directly  across  the  street. 
What  do  you  want  to  do,  LT?” 

Scenario  One  Questions 

1.  Which  is  the  most  likely  munber  of  hostiles  located  in  the  town? 

a.  Less  than  a  squad 

b.  Squad'Size 

c.  Platoon-size 

d.  Company-size 

2.  What  is  the  most  likely  enemy  situation? 

a.  The  enemy  sent  a  unit  to  recon  the  town,  and  they  unexpectedly  found  your  platoon 

b.  A  civilian  faction  in  the  town  has  decided  to  attack  US  forces 

c.  An  enemy  element  is  deliberately  attacking  your  platoon  to  support  their  attack 
against  your  company  in  the  north 

d.  A  small  group  of  enemy  soldiers  avoided  contact  when  1^*  and  2"''  platoons 
cleared  the  town 

3.  What  is  the  most  likely  enemy’s  intent  in  the  town? 

a.  To  prevent  your  unit  from  aiding  the  rest  of  the  company  in  the  firefight  to  the 
north 

b.  To  regain  controi  of  the  town  and  defend  it 

c.  To  seize  your  supplies 

d.  Simply  self-defense 

4.  Which  of  the  following  is  most  likely  to  happen  next? 

a.  3'^''  squad  begins  taking  fire  from  the  house  directiy  across  the  street 

b.  Your  platoon  begins  taking  indirect  fire 

c.  The  hostiles  surrender 

d.  3*^  squad  takes  a  casualty 

5.  After  you’ve  eliminated  the  immediate  threat,  which  is  your  primary  objective? 

a.  Secure  and  occupy  the  grocery  store 

b.  Finish  reloading  supplies  to  prevent  their  theft 

c.  Break  contact  and  move  into  position  to  support  the  company  commander 

d.  Investigate  the  house  directly  across  the  street 
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6.  The  grocery  store  is  a  one-story  building  the  size  of  a  typical  7-Eleven.  Its  walls  are  concrete.  There  are 

front  and  back  entrances;  2  windows  are  located  on  both  the  north  and  south  walls  of  the  store,  and  1 
window  is  located  on  both  the  east  and  west  walls.  If  you  were  to  order  the  store  to  be  cleared  of  the 
hostiles,  how  would  you  divide  your  platoon? 

a.  Three  squads  clear  the  building  while  machine  gunners  provide  support  and  security 

b.  Two  squads  clear  the  building  while  squad  and  machine  gunners  provide  security 

c.  One  squad  clears  the  building  while  two  squads  and  machine  gunners  provide 
security 

d.  One  squad  secures  the  area  with  the  machine  gunners,  one  squad  conducts  a 
breach  of  the  building,  and  one>squad  assaults  and  clears  the  building 

7.  What  immediate  orders  should  you  give  to  your  squad  leaders? 

a.  Cease  fire  and  then  evaluate  the  situation 

b.  Return  fire  and  suppress  the  building  in  order  to  clear  it 

c.  Cease  fire,  maintain  security,  and  finish  loading  supplies 

d.  Abandon  supplies,  suppress  enemy  positions,  and  break  contact  with  the 
enemy  to  get  to  the  rest  of  the  company 

8.  After  contacting  3"^  squad  on  the  radio,  who  do  you  communicate  with  next? 

a.  Platoon  Sergeant 

b.  Try  the  company  commander  again 

c.  1®‘  or  2"*^  platoon  leaders  on  the  company  net 

d.  Switch  to  battalion  net  and  call  the  Main  CP 

Scenario  Two 

You  are  the  platoon  leader  of  3"*  platoon,  A  Company.  Your  company  is  part  of  a  U.N.  task  force 
to  provide  peacekeeping  and  humanitarian  assistance  to  the  war-ravaged  country  of  Rashanna.  Rashaima 
has  several  towns  of  impoverished,  hungry  civilians,  many  of  whom  have  been  left  homeless  by  the  war. 
The  task  force  has  been  in  Rashanna  for  3  months,  and  its  broad  mission  remains  the  same:  Maintain 
order  and  discipline  in  the  country,  and  provide  a  security  presence.  Your  company  is  operating  mainly  in 
three  villages.  The  commander’s  intent  is  as  follows: 

/  want  each  village  tightly  controlled  to  prevent  civilian  casualties.  Ensure  all  weapons  are 
promptly  confiscated  and  destroyed  per  SOP.  Keep  good  intelligence  on  troublemakers,  but 
attempt  to  prevent  provocation.  Evacuate  any  casualties  promptly  to  the  Battalion  Aid  Station. 
Keep  a  special  eye  out  for  demonstration  and  protest  or  any  acts  of  violence  by  the  locals.  If 
disturbance  breaks  out,  bring  to  a  halt  using  the  most  expedient  means. 

Your  platoon  is  currently  located  in  the  town  of  Tralley,  where  conditions  are  similar  to  most 
villages  in  the  country.  The  homes  are  simple  one-stories  constmcted  of  pl5WOod.  Almost  all  have  four 
small  rooms,  each  packed  full  of  residents.  The  street  blocks  have  4  homes  situated  on  each  —  2  per  side, 
backing  the  others.  Separating  the  back  yards  of  the  houses  are  rickety,  6-foot  wooden  fences.  The  local 
marketplace  consists  of  several  small  stands  covered  by  tent-tops,  where  elderly  women  sell  food  and 
clothing. 
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Tralley’s  local  police  force  is  virtually  non-existent  since  order  is  only  now  being  restored  to  the 
region.  Your  platoon  has  been  policing  the  region  -  quelling  disturbances  and  handling  domestic  disputes 
“  while  the  local  force  reorganizes.  You  know  many  of  the  civilians  fairly  well,  and  they  are  familiar  with 
you.  Most  seem  to  appreciate  your  presence,  although  a  few  malcontents  are  not  uncommon.  The  civilians 
have  access  to  a  few  weapons.  On  occasion  you  have  confiscated  a  Molotov  cocktail,  a  grenade,  or  a 
semi-automatic  rifle  from  a  rabble-rouser,  but  so  far  nobody  in  your  platoon  has  been  injured.  Lately, 
however,  civilians  have  been  telling  you  that  a  few  of  the  more  rebellious  townspeople  are  plotting  some 
form  of  uprising  for  El  Dia  Del  Guapo,  the  anniversary  of  the  death  of  the  instigator  of  the  rebellion. 
Given  the  language  barrier,  and  the  way  rumors  are  known  to  spread  in  this  area,  you  are  not  sure  how 
much  stock  to  put  in  this  latest  one. 

Your  platoon  is  conducting  an  on-foot  patrol  of  Tralley  on  the  eve  of  El  Dia  Del  Guapo.  You  are 
with  2”^  squad  walking  north  on  Sosa  Ave.  You  notice  an  unnatural  emptiness  to  the  streets.  Your  platoon 
sergeant  is  up  with  1^^  squad,  50  meters  north  of  your  position.  3”^  squad  is  75  meters  to  your  rear.  Your 
platoon  sergeant  radios  in:  “Sir,  we’ve  just  spotted  four  civilians  carrying  what  looks  like  crates  of 
oranges  from  a  truck  into  the  warehouse  a  block  up.  Two  more  civilians  are  guarding  the  door  of  the 
warehouse.  It  looks  like  whatever  they’re  unloading,  they’re  protecting  it  closely.  From  what  I  can  see 
inside  the  warehouse,  there  are  at  least  three  more  men  in  there.  Some  I  recognize  as  troublemakers.  The 
others  I’ve  never  seen  before.  And  there  are  three  kids  there  playing  marbles  or  something.” 

You  call  the  company  commander  who  is  roughly  3  km  away  in  the  village  of  Daka.  “We’re 
seeing  the  same  thing  going  on  over  here,”  he  tells  you.  “We’ve  confiscated  a  couple  crates  of  grenades 
and  ‘47s.  It’s  getting  hairy  here.  3^^  platoon  had  one  casualty  today.  A  couple  civilians  tell  me  that  at 
sundown  tonight  the  plan  is  for  all  hell  to  break  loose.  You  need  to  clear  that  warehouse  immediately,  and 
stop  things  before  they  start.” 

You  contemplate  how  you’re  going  to  stop  them  while  adhering  to  the  ROE: 

a.  You  have  the  right  to  defend  yourself  against  attacks  or  threats  of  attack. 

b.  U.S.  forces  should  use  the  minimum  force  necessary  under  the  circumstances  and 
proportional  to  the  threat. 

c.  Hostile  fire  may  be  returned  effectively  and  promptly  only  to  promote  the  safety  of 
peacekeeping  forces. 

d.  You  may  seize  the  property  of  others  to  accomplish  your  mission. 

e.  Detention  of  civilians  is  authorized  for  security  reasons  or  for  self-defense. 

f.  Placing  fire  on  religious  or  cultural  centers  or  toward  schools  or  hospitals  is  forbidden. 

Just  then,  another  truck  pulls  up  to  the  warehouse.  It  is  full  of  crates.  Five  men  jump  out  and  immediately 
begin  unloading.  It  is  clear  they  don’t  see  your  platoon  yet,  even  though  1^^  squad  is  only  a  block  away. 
The  time  is  1900,  thirty  minutes  until  sundown. 

Scenario  Two  Questions 

1.  What  is  the  most  likely  intent  of  the  civilians  in  and  around  the  warehouse? 

a.  Instigate  random  mayhem  and  destruction 

b.  Organize  a  rebellion  against  your  platoon  and  the  U.N.  task  force 

c.  Organize  a  faction  and  take  control  of  the  village 

d.  Throw  a  riotous  celebration  of  El  Dia  Del  Guapo 
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2.  What  immediate  orders  should  you  give  to  your  squad  leaders? 

a.  Conceal  your  positions  and  continue  to  watch  the  situation  unfoid 

b.  Each  squad  fire  one  warning  shot 

c.  Continue  your  patrol  with  triggers  ready 

d.  Clear  the  warehouse 

3.  Of  the  following,  what  is  most  likely  to  happen  next? 

a.  Another  truck  arrives  at  the  warehouse 

b.  The  guards  at  the  door  of  the  warehouse  notice  your  platoon 

c.  Large  numbers  of  civilians  approach  and  enter  the  warehouse 

d.  The  civilians  in  the  warehouse  begin  shooting 

4.  Of  the  following,  which  is  your  primary  objective? 

a.  Investigate  the  contents  of  the  crates 

b.  Take  control  of  the  warehouse 

c.  Remove  the  children  from  the  area 

d.  Secure  the  area  surrounding  the  warehouse  to  be  in  position  to  stop  whatever 
violence  might  occur 

5.  If  you  could  have  one  of  these  pieces  of  information,  which  would  help  you  the  most? 

a.  What  1®'  and  3"*  platoons  are  encountering  in  the  villages  they  are  patrolling 

b.  Whether  all  the  civilians  in  and  around  the  warehouse  are  locals  from  Tralley, 
or  whether  some  have  come  from  another  village 

c.  How  Rashannans  typically  act  on  Holidays  of  this  sort 

d.  Whether  ROE  can  change  in  situations  like  this 

6.  Which  course  of  action  is  most  appropriate? 

a.  Take  no  action  unless  the  civilians  initiate  hostile  acts 

b.  Use  force  to  clear  the  warehouse 

c.  One  squad  attempts  to  talk  to  the  people  imloading  the  truck  while  the  other  two  squads 
provide  security 

d.  Take  no  action  until  you  can  get  reinforcements 

7.  As  the  crates  are  imloaded  from  the  truck,  one  falls  and  several  bottles  of  whiskey  scatter  to  the  ground 

as  the  men  frantically  try  to  catch  them.  You  know  that  Rashannans  drink  whiskey  like  Americans 
drink  water.  Based  on  this  additional  information,  what  is  your  assessment  of  the  civilians’  intent? 

a.  Instigate  random  mayhem  and  destruction 

b.  Organize  a  rebellion  against  your  platoon  and  the  U.N.  task  force 

c.  Organize  a  faction  and  take  control  of  the  village 

d.  Throw  a  riotous  celebration  of  El  Dia  Del  Guapo 

8.  The  crate  that  fell  from  the  truck  contained  several  AK-47s  instead  of  bottles  of  whiskey.  Based  on  this 

information,  what  is  your  assessment  of  the  civilians’  intent? 

a.  Instigate  random  mayhem  and  destruction 

b.  Organize  a  rebellion  against  your  platoon  and  the  U.N.  task  force 

c.  Organize  a  faction  and  take  control  of  the  village 

d.  Throw  a  riotous  celebration  of  El  Dia  Del  Guapo 
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SESSION  EVALUATION  FORM 

Please  circle  a  number  on  the  scale  below  to  indicate  the  extent  to  which  you  agree  or 
disagree  with  the  following  statements.  These  statements  all  concern  the  Decision 
Making  Game  (DMG)  session  about  Military  Operations  in  Urban  Terrain  (MOUT)  that 
you  participated  in  today.  Your  answers  are  for  research  purposes  only — ^they  will  not  be 
shared  with  the  facilitator  so  please  give  us  vour  honest  opinion.  Thanks! 


Today’s  date 


Your  name 


6.  Today’s  DMG  session  provided  participants  with  a  valuable  opportunity  to 
practice  their  decision-making  skills. 

1  2  3  4  5 

I - 1 - 1 - 1 - 1 

Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 

7.  The  facilitator  helped  me  think  about  the  many  different  factors  involved  in 
making  rapid  decisions  in  MOUT  settings. 

1  2  3  4  5 

I - 1 - 1 - 1 - 1 

Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 

8.  The  MOUT  scenario  described  in  the  DMG  was  challenging  given  my  current 
knowledge  of  MOUT. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 


9.  The  facilitator  kept  all  of  the  students  actively  involved  during  the  DMG  session. 
1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 


10. 1  would  like  to  participate  in  more  DMG  sessions  in  the  future. 

1  2  3  4  5 


Strongly 

disagree 


Neither  agree 
or  disagree 


Strongly 

agree 


Today’s  date 


Your  name 


CADET  POST-TEST  QUESTIONNAIRE 

Please  circle  a  number  on  the  scale  below  to  indicate  the  extent  to  which  you  agree  or 
disagree  with  the  following  statements.  These  statements  all  concern  the  Decision 
Making  Game  (DMG)  sessions  about  Military  Operations  in  Urban  Terrain  (MOUT)  that 
you  participated  in  during  the  past  few  days. 

1 .  Participation  in  DMG  sessions  should  improve  my  ability  to  make  rapid  decisions 


on  the  battlefield. 

1 

1 

2 

1 

3 

1 

4 

1 _ 

5 

_ 1 

1 - 

Strongly 

1 

1 

Neither  agree 

1 

1 

Strongly 

disagree 

or  disagree 

agree 

2.  I  learned  a  great  deal  of  new  information  about  MOUT  by  participating  in  the 
DMG  sessions. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 

3.  DMG  sessions  allowed  me  to  hear  my  fellow  cadets  describe  the  thinking  that 
went  into  their  solutions  to  the  DMGs. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 


4.  DMG  sessions  provide  cadets  with  valuable  opportunities  to  practice  their 
decision-making  skills. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 

5.  Participating  in  the  DMGs  sessions  made  me  diink  about  the  many  different 
factors  involved  in  making  rapid  decisions  in  MOUT  settings. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 
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6.  Overall,  the  MOUT  scenarios  described  in  the  DMGs  were  challenging  given  my 
current  knowledge  of  MOUT. 

1  2  3  4  5 


Strongly 

disagree 


Neither  agree 
or  disagree 


Strongly 

agree 


7.  I  would  like  to  participate  in  more  DMG  sessions  in  the  future. 


Strongly 

disagree 


Neither  agree 
or  disagree 


Strongly 

agree 


8.  DMG  sessions  demonstrated  that  other  cadets  can  look  at  the  same  situation  and 
come  up  with  very  different  (but  reasonable)  courses  of  action. 

1  2  3  4  5 

I - 1 - 1 - 1 - 1 


Strongly 

disagree 


Neither  agree 
or  disagree 


Strongly 

agree 


9.  DMG  sessions  demonstrated  that  the  decisions  platoon  leaders  have  to  make  in 
MOUT  situations  involve  a  surprising  number  of  subtle  judgments. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 

10.  Participation  in  the  DMG  sessions  should  improve  my  ability  to  communicate 
orders  to  my  subordinates. 


1 

1 

2 

1 

3 

1 

4 

5 

_ 1 

1 - 

Strongly 

1 

1 

Neither  agree 

1 

Strongly 

disagree 

or  disagree 

agree 
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FACILITATOR  POST-SESSION  QUESTIONNAIRE— NO-IMPACT 


This  information  is  for  research  purposes  only  so  please  give  us  yom  honest  opinion. 
Thank  you  for  yom  participation. 

Part  I.  Please  circle  a  number  on  the  scale  below  to  indicate  the  extent  to  which  you  agree 
or  disagree  with  the  following  statements. 

1 .  The  DMG  sessions  I  conducted  were  valuable  learning  experiences  for  the  cadets 


who  participated. 

1  2 

1  1 

3 

1 

4  5 

1 _ 1 

1 - 1 

Strongly 

1 

Neither  agree 

1  \ 

Strongly 

disagree 

or  disagree 

agree 

Participation  m  DMG  sessions  like  the  ones  I  conducted  should  improve  the 
participants’  ability  to  make  rapid  decisions  on  the  battlefield. 

1  2 

1  1 

3 

1  _ 

4  5 

1 _ 1 

1 - \ 

Strongly 

1 

Neither  agree 

1  1 

Strongly 

disagree 

or  disagree 

agree 

DMG  sessions  provide  participants  with  valuable  opportunities  to  practice  their 

communication  skills. 

1  2 

1  1 

3 

1 

4  5 

_ 1 _ 1 

1 - 1 

Strongly 

1 

Neither  agree 

1  1 

Strongly 

disagree 

or  disagree 

agree 

DMG  sessions  provide  cadets  with  valuable  opportunities  to  practice  their 

decision-making  skills. 

1  2 

1  1 

3 

1 

4  5 

1 _ 1 

1 - 1 

Strongly 

1 

Neither  agree 

1  1 

Strongly 

disagree 

or  disagree 

agree 

Facilitating  these  DMG  sessions  made  me  think  about  the  many  different  factors 
involved  in  making  rapid  decisions  in  MOUT  settings. 

1  2 

1  1 

1 

4  5 

1 _ 1 

1 - 1 

Strongly 

1 

Neither  agree 

1  1 

Strongly 

disagree 

or  disagree 

agree 
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6.  The  DMG  sessions  I  facilitated  provided  a  great  context  for  me  to  share  my 
knowledge  of  battlefield  tactics  with  a  group  of  cadets. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 

7.  As  the  facilitator,  I  was  able  to  keep  all  of  the  cadets  actively  involved  during  the 
DMG  sessions. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  asree 

8.  As  the  facilitator,  I  was  able  to  focus  the  group’s  discussion  on  the  critical  factors 
that  influence  rapid  decision  making  in  battlefield  settings. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 

9.  lam  likely  to  use  DMGs  in  the  future  to  improve  the  decision-making  skill  of  my 
subordinates. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree 


10.  The  DMG  sessions  demonstrated  that  different  cadets  can  look  at  the  same 
situation  and  come  up  with  very  different  (but  reasonable)  courses  of  action. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 

1 1. 1  would  have  been  more  confident  facilitating  DMG  sessions  focused  on 
traditional  battlefield  scenarios  instead  of  MOUT  scenarios. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 
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Part  II.  Please  provide  your  answers  to  the  following  questions  in  the  space  provided.  If 
need  be,  you  can  continue  your  answer  on  the  back  of  this  form. 

1 .  How  much  time  did  you  spend  preparing  for  the  first  DMG  session? 


2.  How  much  time  did  you  spend  preparing  for  the  second  DMG  session? 


3.  What  could  we  have  done  to  better  prepare  you  for  your  role  as  the  facilitator  for 
these  DMG  sessions? 


4.  What  suggestions  for  improvements  do  you  have  for  the  DMG  scenario  used  for 
the  first  DMG  session  you  facilitated? 
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5.  What  suggestions  for  improvements  do  you  have  for  the  DMG  scenario  used 
the  second  DMG  session  you  facilitated? 


6.  Do  you  have  any  other  suggestions  or  comments? 


THANK  YOU  FOR  PARTICIPATING! 


FACILITATOR  POST-SESSION  QUESTIONNAIRE— IMPACT 


This  information  is  for  research  purposes  only  so  please  give  us  your  honest  opinion. 
Thank  you  for  your  participation. 

Part  I.  Please  circle  a  niunber  on  the  scale  below  to  indicate  the  extent  to  which  you  agree 
or  disagree  with  the  following  statements. 

1 .  The  DMG  sessions  I  conducted  were  valuable  learning  experiences  for  the  cadets 
who  participated. 


1 

1 

2 

1 

3  4 

1  1 

5 

_ 1 

1 

Strongly 

1 

1  1 

Neither  agree 

1 

Strongly 

disagree 

or  disagree 

agree 

Participation  in  DMG  sessions  like  the  ones  I  conducted  should  improve  the 

participants’  ability  to  make  rapid  decisions  on  the  battlefield. 

1 

1 

2 

1 

3  4 

1  1 

5 

1 

1 - 

Strongly 

1 

1  1 

Neither  agree 

1 

Strongly 

disagree 

or  disagree 

agree 

DMG  sessions  provide  participants  with  valuable  opportunities  to  practice  their 

communication  skills. 

1 

1 

2 

\ 

3  4 

1  1 

5 

1 - 

Strongly 

1 

1  i 

Neither  agree 

Strongly 

disagree 

or  disagree 

agree 

DMG  sessions  provide  cadets  with  valrxable  opportimities  to  practice  their 

decision-making  skills. 

1 

1 

2 

1 

3  4 

1  1 

5 

_ 1 

1 - 

Strongly 

1 

1  1 

Neither  agree 

1 

Strongly 

disagree 

or  disagree 

agree 

Facilitating  these  DMG  sessions  made  me  think  about  the  many  different  factors 

involved  in  making  rapid  decisions  in  MOUT  settings. 

1 

1 

2 

1 

3  4 

1  1 

5 

1 

Strongly 

1 

1  i 

Neither  agree 

Strongly 

disagree 

or  disagree 

agree 
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6.  The  DMG  sessions  I  facilitated  provided  a  great  context  for  me  to  share  my 
knowledge  of  battlefield  tactics  with  a  group  of  cadets. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 

7.  As  the  facilitator,  I  was  able  to  keep  all  of  the  cadets  actively  involved  during  the 
DMG  sessions. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  ^gT^e 

8.  As  the  facilitator,  I  was  able  to  focus  the  group’s  discussion  on  the  critical  factors 
that  influence  rapid  decision  making  in  battlefield  settings. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 

9.  I  am  likely  to  use  DMGs  in  the  future  to  improve  the  decision-making  skill  of  my 
subordinates. 

5 

- 1 

Strongly 
agree 

10.  The  DMG  sessions  demonstrated  that  different  cadets  can  look  at  the  same 
situation  and  come  up  with  very  different  (but  reasonable)  courses  of  action. 

1  2  3  4  5 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 

1 1. 1  would  have  been  more  confident  facilitating  DMG  sessions  focused  on 
traditional  battlefield  scenarios  instead  of  MOUT  scenarios. 

1  2  3  4  5 


Strongly 

disagree 


Neither  agree 
or  disagree 


Strongly  Neither  agree  Strongly 

disagree  or  disagree  agree 
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Part  II.  Please  provide  your  answers  to  the  following  questions  in  the  space  provided.  If 
need  be,  you  can  continue  your  answer  on  the  back  of  this  form. 

1 .  How  much  time  did  you  spend  reviewing  the  IMPACT  CD  before  you  facilitated 
the  first  DMG  session? 


2.  Did  you  do  anything  else  (besides  reviewing  the  IMPACT  CD)  in  order  to  prepare 
for  the  first  DMG  session?  If  so,  please  describe  what  you  did  and  about  how  long 
you  spent  on  this  preparation. 


3.  How  much  time  (if  any)  did  you  spend  reviewing  the  IMPACT  CD  after  you 
facilitated  the  first  DMG  session  and  before  you  facilitated  the  second  DMG 
session? 


4.  Did  you  do  anything  else  (besides  reviewing  the  IMPACT  CD)  in  order  to  prepare 
for  the  second  DMG  session?  If  so,  please  describe  what  you  did  and  about  how 
long  you  spent  on  this  preparation. 
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5.  What  could  we  have  done  to  better  prepare  you  for  your  role  as  the  facilitator  for 
these  DMG  sessions? 


6.  What  suggestions  for  improvements  do  you  have  for  the  DMG  scenario  used  for 
the  first  DMG  session  you  facilitated? 


7.  What  suggestions  for  improvements  do  you  have  for  the  DMG  scenario  used  for 
the  second  DMG  session  you  facilitated? 


8.  On  the  scales  below,  please  rate  the  usefulness  of  the  different  modules  provided 
on  the  IMPACT  CD  for  preparing  you  to  facilitate  the  DMG  sessions. 


A.  Introduction 

1  2 

1  1 

3 

1 

4 

1 

5 

_ 1 

NA 

1 - 1 

Not  at  all 

1 

Somewhat 

1 

1 

Extremely 

I  didn’t 

useful 

B.  Facilitating  DMGs 

useful 

useful 

review  this 
module 

1  2 

1  1 

3 

1  --  - 

4 

1 

5 

_ 1 

NA 

1  1 

Not  at  all 

1 

Somewhat 

1 

1 

Extremely 

I  didn’t 

useful 

useful 

useful 

review  this 
module 
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C.  Decision  Making  Critique  (DMC) 


1 

1 

2 

1 

3 

1 

4 

1 

5 

_ 1 

NA 

1 - 

Not  at  all 
usefiil 

1 

Somewhat 

usefiil 

1 

1 

Extremely 

useful 

I  didn’t 
review  this 
module 

D.  Creating  DMGs 

1 

1 

2 

1 

3 

1 

4 

1 

5 

_ 1 

NA 

1 - 

Not  at  all 
useful 

1 

Somewhat 

usefiil 

1 

1 

Extremely 

usefiil 

I  didn’t 
review  this 
module 

E.  DMG  library 

1 

1 

2 

1 

3 

1 

4 

1  _  . 

5 

_ 1 

NA 

1 - 

Not  at  all 
useful 

— i 

1 

Somewhat 

usefiil 

1 

1 

Extremely 

usefiil 

I  didn’t 
review  this 
module 

9.  How  useful  was  the  Decision  Making  Critique  (DMC)  when  facilitating  the  DMG 
sessions  with  the  cadets?  Check  the  answer  the  best  describes  your  use  of  the 
DMC. 

_  I  didn’t  try  to  use  the  DMC 

_ I  used  a  similar  approach,  but  did  not  use  any  of  the  specific  questions 

provided  in  the  DMC 

_ used  most  of  the  questions  on  the  DMC 

10.  How  similar  is  the  Decision  Making  Critique  to  a  standard  After  Action  Review? 

1  2  3  4  5 


Not  at  all  Somewhat  Same 

similar  similar 

1 1 .  How  likely  are  you  to  incorporate  elements  of  the  Decision  Making  Critique  in 
future  After  Action  Reviews  you  lead? 

1  2  3  4  5 


Not  at  all 
likely 


Somewhat 

likely 
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12.  What  do  you  think  the  key  differences  are  (if  any)  between  the  Decision  Making 
Critique  (DMC)  and  the  traditional  After  Action  review  (AAR)? 

13.  What  did  you  like  best  about  the  IMPACT  CD?  Why? 

14.  What  part  of  the  IMPACT  CD  did  you  like  the  least?  Why? 

15.  Do  you  have  any  other  suggestions  or  comments?  [About  the  IMPACT  CD  or  the 
DMG  sessions] 

THANK  YOU  FOR  PARTICIPATING! 
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